Sep 09 10:24:08.501143 ip-10-0-131-250 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 10:24:08.501152 ip-10-0-131-250 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 10:24:08.501159 ip-10-0-131-250 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 10:24:08.501344 ip-10-0-131-250 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 10:24:18.670289 ip-10-0-131-250 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 10:24:18.670308 ip-10-0-131-250 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9538f3c04d8f4f6ebb577f9f00d6ae35 --
Sep 09 10:25:56.241504 ip-10-0-131-250 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 10:25:56.648436 ip-10-0-131-250 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 10:25:56.648436 ip-10-0-131-250 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 10:25:56.648436 ip-10-0-131-250 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 10:25:56.648436 ip-10-0-131-250 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 10:25:56.648436 ip-10-0-131-250 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 10:25:56.649596 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.649505    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 10:25:56.653806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653792    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:25:56.653806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653806    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653809    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653813    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653816    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653818    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653821    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653824    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653827    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653829    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653832    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653835    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653839    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653842    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653845    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653850    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653854    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653857    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653860    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653864    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653867    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:25:56.653871 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653870    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653874    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653878    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653881    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653884    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653886    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653889    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653892    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653895    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653898    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653900    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653903    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653905    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653908    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653911    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653914    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653916    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653919    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653921    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:25:56.654357 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653924    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653927    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653929    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653932    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653935    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653938    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653940    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653943    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653946    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653948    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653950    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653953    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653955    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653960    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653963    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653966    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653970    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653973    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653976    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:25:56.654862 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653980    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653982    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653985    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653988    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653990    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653993    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653995    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.653998    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654000    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654003    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654005    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654008    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654010    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654013    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654015    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654018    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654020    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654023    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654026    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654028    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:25:56.655348 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654031    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654034    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654037    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654039    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654042    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654044    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654047    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654049    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654053    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654056    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654058    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654061    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654064    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654067    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654069    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654071    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654074    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654076    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654079    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654081    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:25:56.655823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654865    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654872    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654876    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654879    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654882    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654886    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654889    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654891    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654894    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654896    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654899    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654907    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654911    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654915    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654918    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654922    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654926    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654928    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654931    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:25:56.656335 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654933    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654937    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654940    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654943    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654945    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654948    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654951    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654953    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654956    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654958    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654961    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654964    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654966    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654969    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654972    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654974    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654977    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654980    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654982    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654985    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:25:56.656805 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654988    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654990    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654992    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654995    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.654998    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655006    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655009    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655011    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655014    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655017    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655020    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655022    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655025    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655028    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655031    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655033    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655036    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655038    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655041    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655045    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:25:56.657343 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655047    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655050    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655052    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655054    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655057    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655059    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655062    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655065    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655067    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655070    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655072    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655075    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655077    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655080    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655082    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655085    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655088    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655090    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655098    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655101    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:25:56.657829 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655104    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655109    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655111    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655114    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655117    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655120    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655122    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655125    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655127    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655129    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655132    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655135    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655137    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655140    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655142    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655144    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655147    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655149    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655152    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655155    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:25:56.658331 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655559    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655565    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655568    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655570    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655573    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655575    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655578    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655581    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655584    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655586    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655590    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655593    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655595    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655600    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655603    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655605    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655609    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655613    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655616    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655619    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:25:56.658823 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655622    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655624    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655627    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655629    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655632    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655635    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655637    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655640    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655642    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655645    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655647    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655650    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655652    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655655    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655657    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655662    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655666    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655669    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655672    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:25:56.659340 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655676    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655679    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655682    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655684    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655687    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655690    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655694    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655697    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655700    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655702    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655705    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655707    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655710    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655713    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655716    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655718    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655721    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655723    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655725    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:25:56.659806 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655730    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655733    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655736    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655739    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655741    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655744    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655747    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655749    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655752    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655754    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655757    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655759    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655761    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655765    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655768    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655770    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655772    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655775    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655778    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655783    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:25:56.660295 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655786    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655788    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655791    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655793    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655796    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655798    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655801    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655804    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655806    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655809    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655811    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655814    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655817    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655819    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655821    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655824    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655827    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655830    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655833    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655836    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.655838    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:25:56.660784 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655911    2586 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655955    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655961    2586 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655965    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655970    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655973    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655977    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655981    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655985    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655987    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655992    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655995    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.655998    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656001    2586 flags.go:64] FLAG: --cgroup-root=""
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656004    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656007    2586 flags.go:64] FLAG: --client-ca-file=""
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656010    2586 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656012    2586 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656017    2586 flags.go:64] FLAG: --cluster-domain=""
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656020    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656023    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656026    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656029    2586 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656033    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 10:25:56.661380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656036    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656039    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656042    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656045    2586 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656048    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656051    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656054    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656057    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656061    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656064    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656066    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656069    2586 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656072    2586 flags.go:64] FLAG: --enable-server="true"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656075    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656079    2586 flags.go:64] FLAG: --event-burst="100"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656082    2586 flags.go:64] FLAG: --event-qps="50"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656085    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656087    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656091    2586 flags.go:64] FLAG: --eviction-hard=""
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656095    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656098    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656101    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656104    2586 flags.go:64] FLAG: --eviction-soft=""
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656107    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656110    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 10:25:56.661992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656112    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656115    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656118    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656121    2586 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656124    2586 flags.go:64] FLAG: --feature-gates=""
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656127    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656130    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656133    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656136    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656139    2586 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656142    2586 flags.go:64] FLAG: --help="false"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656145    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-131-250.ec2.internal"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656148    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656151    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656154    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656158    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656161    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656177    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656180    2586 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656183    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656186    2586 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656189    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656192    2586 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656195    2586 flags.go:64] FLAG: --kube-reserved=""
Sep 09 10:25:56.662668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656198    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656201    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656205    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656208    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656211    2586 flags.go:64] FLAG: --lock-file=""
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656214    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656217    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656220    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656225    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656228    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656230    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656233    2586 flags.go:64] FLAG: --logging-format="text"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656240    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656244    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656247    2586 flags.go:64] FLAG: --manifest-url=""
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656250    2586 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656254    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656257    2586 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656261    2586 flags.go:64] FLAG: --max-pods="110"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656265    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656268    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656271    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656274    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656277    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656279    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 10:25:56.663279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656282    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656289    2586 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656292    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656295    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656298    2586 flags.go:64] FLAG: --pod-cidr=""
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656301    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656306    2586 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656309    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656312    2586 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656315    2586 flags.go:64] FLAG: --port="10250"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656318    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656322    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04f464d49cd08cbc9"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656325    2586 flags.go:64] FLAG: --qos-reserved=""
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656328    2586 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656331    2586 flags.go:64] FLAG: --register-node="true"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656334    2586 flags.go:64] FLAG: --register-with-taints=""
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656337    2586 flags.go:64] FLAG: --registry-burst="10"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656340    2586 flags.go:64] FLAG: --registry-qps="5"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656343    2586 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656346    2586 flags.go:64] FLAG: --reserved-memory=""
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656350    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656354    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656357    2586 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656360    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656363    2586 flags.go:64] FLAG: --runonce="false"
Sep 09 10:25:56.663900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656366    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656369    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656372    2586 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656375    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656378    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656381    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656384    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656387    2586 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656389    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656392    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656395    2586 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656398    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656401    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656404    2586 flags.go:64] FLAG: --system-cgroups=""
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656406    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656414    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656416    2586 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656420    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656424    2586 flags.go:64] FLAG: --tls-min-version=""
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656427    2586 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656430    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656433    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656436    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656438    2586 flags.go:64] FLAG: --v="2"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656443    2586 flags.go:64] FLAG: --version="false"
Sep 09 10:25:56.664577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656447    2586 flags.go:64] FLAG: --vmodule=""
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656450    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.656453    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656570    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656574    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656577    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656580    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656583    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656586    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656588    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656591    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656594    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656596    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656599    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656601    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656605    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656607    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656609    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656612    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656615    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:25:56.665449 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656617    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656620    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656624    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656626    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656629    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656631    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656634    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656637    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656639    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656642    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656645    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656647    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656650    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656653    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656656    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656661    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656663    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656666    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656670    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:25:56.666068 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656673    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656676    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656678    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656681    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656684    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656686    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656689    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656691    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656694    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656696    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656699    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656702    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656704    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656707    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656709    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656713    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656716    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656719    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656722    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656725    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:25:56.666575 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656727    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656730    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656733    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656735    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656738    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656740    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656743    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656745    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656749    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656752    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656754    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656757    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656759    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656762    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656764    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656767    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656769    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656772    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656774    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656777    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:25:56.667104 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656779    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656782    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656786    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656789    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656792    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656795    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656798    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656802    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656805    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656808    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656811    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656814    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656817    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656820    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656823    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656826    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656830    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656833    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656836    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:25:56.667628 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656839    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656843    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656846    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.656848    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.657516    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.666246    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.666262    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.666282    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.666292    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666350    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666355    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666358    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666362    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666365    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666368    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:25:56.668097 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666371    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666376    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666379    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666381    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666384    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666387    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666390    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666393    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666396    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666399    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666401    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666404    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666407    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666410    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666413    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666415    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666418    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666425    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666428    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666431    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:25:56.668495 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666434    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666436    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666439    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666443    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666446    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666448    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666451    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666454    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666457    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666460    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666463    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666466    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666468    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666471    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666474    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666476    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666479    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666481    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666491    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666494    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:25:56.668968 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666497    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666500    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666503    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666506    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666508    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666511    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666514    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666517    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666520    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666522    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666525    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666527    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666530    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666534    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666538    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666541    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666543    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666546    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666549    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:25:56.669481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666552    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666556    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666560    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666563    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666567    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666570    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666574    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666577    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666580    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666582    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666585    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666588    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666591    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666593    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666597    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666600    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666602    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666605    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:25:56.669950 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666608    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666610    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666613    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666615    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666618    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666620    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666623    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666626    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666628    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666631    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666634    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666637    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666640    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666642    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666645    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:25:56.670431 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666648    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.666653    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666776    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666782    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666785    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666787    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666790    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666793    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666796    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666798    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666801    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666804    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666806    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666809    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666812    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 10:25:56.670815 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666815    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666818    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666820    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666823    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666825    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666828    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666830    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666833    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666835    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666838    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666840    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666843    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666846    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666849    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666852    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666854    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666857    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666859    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666862    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666865    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 10:25:56.671200 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666867    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666870    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666872    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666875    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666878    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666880    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666883    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666886    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666889    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666891    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666893    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666896    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666899    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666902    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666905    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666907    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666910    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666912    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666915    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666917    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 10:25:56.671682 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666920    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666923    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666925    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666927    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666930    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666933    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666935    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666938    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666940    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666943    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666946    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666948    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666951    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666954    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666956    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666958    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666961    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666963    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666966    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666970    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 10:25:56.672238 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666973    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666976    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666979    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666982    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666984    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666987    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666990    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666994    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.666997    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667000    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667002    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667004    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667008    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667012    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667015    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667017    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667020    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667023    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667025    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 10:25:56.672715 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667028    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667030    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667033    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667035    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667038    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667040    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:56.667043    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.667048    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.667200    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.669734    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.670788    2586 server.go:1019] "Starting client certificate rotation"
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.670882    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 10:25:56.673204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.670944    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 10:25:56.673534 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.673521    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 10:25:56.675499 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.675481    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 10:25:56.686904 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.686884    2586 log.go:25] "Validated CRI v1 runtime API"
Sep 09 10:25:56.691214 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.691201    2586 log.go:25] "Validated CRI v1 image API"
Sep 09 10:25:56.692024 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.692011    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 10:25:56.694666 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.694640    2586 fs.go:135] Filesystem UUIDs: map[0d668726-fecb-4858-ba85-59ad476e7fac:/dev/nvme0n1p3 757bea3a-dbc2-430f-946f-6a57c7b7034c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 10:25:56.694734 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.694665    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 10:25:56.701888 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.701763    2586 manager.go:216] Machine: {Timestamp:2026-09-09 10:25:56.700388145 +0000 UTC m=+0.361672813 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091562 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20ba645c9345a6c473375ec7a9bbb7 SystemUUID:ec20ba64-5c93-45a6-c473-375ec7a9bbb7 BootID:9538f3c0-4d8f-4f6e-bb57-7f9f00d6ae35 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:63:a4:0b:8d:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:63:a4:0b:8d:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:c4:ea:d7:0c:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 10:25:56.701888 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.701875    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 10:25:56.702028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.701965    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 10:25:56.703411 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.703264    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 10:25:56.703810 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.703786    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 10:25:56.703980 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.703819    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 10:25:56.704715 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.704703    2586 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 10:25:56.704768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.704721    2586 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 10:25:56.704768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.704732    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 10:25:56.704768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.704758    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 10:25:56.706672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.706659    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 10:25:56.711419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.711405    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 10:25:56.711562 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.711551    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 10:25:56.711629 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.711587    2586 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 10:25:56.711629 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.711607    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 10:25:56.711629 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.711626    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 10:25:56.711752 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.711641    2586 kubelet.go:398] "Adding apiserver pod source"
Sep 09 10:25:56.711752 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.711672    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 10:25:56.714393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.714341    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 10:25:56.714609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.714596    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 10:25:56.717623 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.717607    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 10:25:56.720600 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.720585    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 10:25:56.720657 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.720615    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 10:25:56.720790 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.720779    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 10:25:56.720831 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.720794    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 10:25:56.720831 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.720800    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 10:25:56.720831 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.720805    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 10:25:56.720831 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.720813    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 10:25:56.720831 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.720818    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 10:25:56.721531 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.721518    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 10:25:56.721531 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.721528    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 10:25:56.721531 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.721535    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 10:25:56.721660 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.721540    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 10:25:56.721660 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.721549    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 10:25:56.721660 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.721557    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 10:25:56.722324 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.722314    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 10:25:56.722361 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.722325    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 10:25:56.723504 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.723461    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 10:25:56.723597 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.723576    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 10:25:56.726396 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.726383    2586 server.go:1268] "Started kubelet"
Sep 09 10:25:56.726494 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.726433    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 10:25:56.726547 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.726476    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 10:25:56.726592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.726551    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 10:25:56.727260 ip-10-0-131-250 systemd[1]: Started Kubernetes Kubelet.
Sep 09 10:25:56.728188 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.728157    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 10:25:56.728380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.728363    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 10:25:56.729419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.729405    2586 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 10:25:56.733587 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.733571    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 10:25:56.736060 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.736044    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 10:25:56.736762 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.736744    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 10:25:56.736925 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.736891    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:56.737593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737573    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 10:25:56.737593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737578    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 10:25:56.737716 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737603    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 10:25:56.737716 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737622    2586 factory.go:55] Registering systemd factory
Sep 09 10:25:56.737716 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737676    2586 factory.go:223] Registration of the systemd container factory successfully
Sep 09 10:25:56.737848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737730    2586 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 10:25:56.737848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737740    2586 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 10:25:56.737944 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737896    2586 factory.go:153] Registering CRI-O factory
Sep 09 10:25:56.737944 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737909    2586 factory.go:223] Registration of the crio container factory successfully
Sep 09 10:25:56.738038 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737952    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 10:25:56.738038 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737966    2586 factory.go:103] Registering Raw factory
Sep 09 10:25:56.738038 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.737995    2586 manager.go:1196] Started watching for new ooms in manager
Sep 09 10:25:56.738314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.738301    2586 manager.go:349] Starting recovery of all containers
Sep 09 10:25:56.738550 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.737329    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3a024c1c45ec0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 10:25:56.726357696 +0000 UTC m=+0.387642367,LastTimestamp:2026-09-09 10:25:56.726357696 +0000 UTC m=+0.387642367,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 10:25:56.740494 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.740464    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 10:25:56.741546 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.741521    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 10:25:56.751782 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.751763    2586 manager.go:354] Recovery completed
Sep 09 10:25:56.757368 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.757356    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:25:56.759144 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.759124    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:25:56.759244 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.759178    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:25:56.759244 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.759194    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:25:56.759827 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.759803    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 10:25:56.759827 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.759813    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:25:56.759827 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.759818    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 10:25:56.759990 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.759834    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 10:25:56.760571 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.760554    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:25:56.760630 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.760586    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:25:56.760630 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.760599    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:25:56.762030 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.762011    2586 policy_none.go:49] "None policy: Start"
Sep 09 10:25:56.762030 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.762034    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 10:25:56.762234 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.761961    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3a024c3b8aef9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-250.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 10:25:56.759146233 +0000 UTC m=+0.420430906,LastTimestamp:2026-09-09 10:25:56.759146233 +0000 UTC m=+0.420430906,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 10:25:56.762234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.762049    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 10:25:56.764224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.764210    2586 policy_none.go:47] "Start"
Sep 09 10:25:56.770770 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.770699    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3a024c3b9487d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-250.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 10:25:56.759185533 +0000 UTC m=+0.420470206,LastTimestamp:2026-09-09 10:25:56.759185533 +0000 UTC m=+0.420470206,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 10:25:56.773937 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.773869    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3a024c3b98053  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-250.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 10:25:56.759199827 +0000 UTC m=+0.420484499,LastTimestamp:2026-09-09 10:25:56.759199827 +0000 UTC m=+0.420484499,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 10:25:56.784289 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.784226    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-250.ec2.internal.18d3a024c3b8aef9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3a024c3b8aef9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-250.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 10:25:56.759146233 +0000 UTC m=+0.420430906,LastTimestamp:2026-09-09 10:25:56.760571284 +0000 UTC m=+0.421855961,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.788569    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-250.ec2.internal.18d3a024c3b9487d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3a024c3b9487d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-250.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 10:25:56.759185533 +0000 UTC m=+0.420470206,LastTimestamp:2026-09-09 10:25:56.760592617 +0000 UTC m=+0.421877290,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.792605    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-250.ec2.internal.18d3a024c3b98053\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3a024c3b98053  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-250.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 10:25:56.759199827 +0000 UTC m=+0.420484499,LastTimestamp:2026-09-09 10:25:56.760604859 +0000 UTC m=+0.421889532,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.796417    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dvdmt"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.798697    2586 manager.go:337] "Starting Device Plugin manager"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.798728    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.798737    2586 server.go:87] "Starting device plugin registration server"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.799081    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.799093    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.799241    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.799343    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.799353    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.800215    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.800263    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.803104    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3a024c634d0bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 10:25:56.800835775 +0000 UTC m=+0.462120434,LastTimestamp:2026-09-09 10:25:56.800835775 +0000 UTC m=+0.462120434,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 10:25:56.809058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.805528    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dvdmt"
Sep 09 10:25:56.860951 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.860923    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 10:25:56.862109 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.862083    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 10:25:56.862229 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.862141    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 10:25:56.862229 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.862188    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 10:25:56.862332 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.862236    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 10:25:56.874121 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.874105    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:25:56.899769 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.899684    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:25:56.901679 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.901657    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:25:56.901761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.901700    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:25:56.901761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.901714    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:25:56.901761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.901741    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-250.ec2.internal"
Sep 09 10:25:56.916074 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.916056    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-250.ec2.internal"
Sep 09 10:25:56.916179 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.916077    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-250.ec2.internal\": node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:56.960053 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.960024    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:56.962973 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.962336    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"]
Sep 09 10:25:56.963069 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.963028    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:25:56.963981 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.963967    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:25:56.964049 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.963994    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:25:56.964049 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.964004    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:25:56.966383 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.966371    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:25:56.966507 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.966491    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:56.966552 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.966524    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:25:56.967082 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.967066    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:25:56.967134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.967095    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:25:56.967134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.967106    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:25:56.967237 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.967068    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:25:56.967237 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.967181    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:25:56.967237 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.967196    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:25:56.969361 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.969345    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:56.969455 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.969368    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:25:56.971743 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.971722    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:25:56.971838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.971750    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:25:56.971838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:56.971762    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:25:56.982766 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.982736    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-250.ec2.internal\" not found" node="ip-10-0-131-250.ec2.internal"
Sep 09 10:25:56.988262 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:56.988246    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-250.ec2.internal\" not found" node="ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.038957 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.038940    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.039058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.038966    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.039058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.038988    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918f93a5380fe0ffc07a388ced93b00-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.ec2.internal\" (UID: \"b918f93a5380fe0ffc07a388ced93b00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.060139 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:57.060117    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:57.139325 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.139297    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.139455 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.139366    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.139455 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.139391    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918f93a5380fe0ffc07a388ced93b00-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.ec2.internal\" (UID: \"b918f93a5380fe0ffc07a388ced93b00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.139455 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.139423    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.139455 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.139436    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918f93a5380fe0ffc07a388ced93b00-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.ec2.internal\" (UID: \"b918f93a5380fe0ffc07a388ced93b00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.139590 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.139471    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.160650 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:57.160597    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:57.261370 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:57.261335    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:57.285512 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.285481    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:57.291602 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:57.291585    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:58.967626 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:58.967590    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.068121 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.068052    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.168617 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.168594    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.221472 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.221446    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:25:59.269651 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.269625    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.276780 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.276765    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 10:25:59.276912 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.276878    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 10:25:59.276960 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.276926    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 10:25:59.370702 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.370676    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.412043 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.411997    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 10:25:59.412987 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.412970    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 10:25:59.413094 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.413007    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 10:25:59.413094 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.413031    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 10:25:59.415106 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.415083    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 10:20:58 +0000 UTC" deadline="2028-04-16 18:08:33.872811566 +0000 UTC"
Sep 09 10:25:59.415106 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.415106    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14047h42m34.45770893s"
Sep 09 10:25:59.470902 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.470863    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.571406 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.571379    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.646703 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:25:59.646537    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb918f93a5380fe0ffc07a388ced93b00.slice/crio-82910571ca4367669fc3b0b00e9406245c1b82e73a1bf728b8e119b5351b1f7c WatchSource:0}: Error finding container 82910571ca4367669fc3b0b00e9406245c1b82e73a1bf728b8e119b5351b1f7c: Status 404 returned error can't find the container with id 82910571ca4367669fc3b0b00e9406245c1b82e73a1bf728b8e119b5351b1f7c
Sep 09 10:25:59.651643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.651627    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:25:59.671649 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.671628    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.685819 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.685793    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:25:59.772291 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.772252    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.873038 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:25:59.873005    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 10:25:59.919239 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.919188    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:25:59.943167 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.943146    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:59.960457 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.960436    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 10:25:59.961195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.961183    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 10:25:59.981761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:25:59.981744    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 10:26:00.231195 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.231161    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb08136825eb2ae62fa73ecf8896a9e.slice/crio-9027bde1ed9d353d460bf927a062978f7f30b829d6c327bc02dd13a16b5d0148 WatchSource:0}: Error finding container 9027bde1ed9d353d460bf927a062978f7f30b829d6c327bc02dd13a16b5d0148: Status 404 returned error can't find the container with id 9027bde1ed9d353d460bf927a062978f7f30b829d6c327bc02dd13a16b5d0148
Sep 09 10:26:00.319304 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.319269    2586 apiserver.go:52] "Watching apiserver"
Sep 09 10:26:00.325604 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.325579    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 10:26:00.325982 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.325963    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-sdmdp","openshift-ovn-kubernetes/ovnkube-node-bhdm7","kube-system/global-pull-secret-syncer-4bqkk","kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal","openshift-dns/node-resolver-flzz6","openshift-image-registry/node-ca-dl5gz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal","openshift-multus/multus-additional-cni-plugins-gjvjf","openshift-multus/multus-s5x6h","openshift-multus/network-metrics-daemon-d48rq","kube-system/konnectivity-agent-mgvw6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c","openshift-cluster-node-tuning-operator/tuned-mh9h5","openshift-network-diagnostics/network-check-target-x2jbl"]
Sep 09 10:26:00.328697 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.328680    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.331687 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.331665    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.331792 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.331762    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-txrf8\""
Sep 09 10:26:00.331860 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.331798    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 10:26:00.333368 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.333350    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:26:00.333906 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.333869    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.333992 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.333961    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:00.336036 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.336016    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.338231 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.338213    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.340408 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.340390    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.342438 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.342420    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 10:26:00.342619 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.342602    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:00.342710 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.342688    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:00.343030 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343015    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 10:26:00.343189 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343173    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 10:26:00.343284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343200    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 10:26:00.343284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343220    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 10:26:00.343486 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343469    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 10:26:00.343557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343470    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 10:26:00.343557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343490    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 10:26:00.343557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343549    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 10:26:00.343699 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343618    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 10:26:00.343699 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343656    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 10:26:00.343803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343698    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t9q8x\""
Sep 09 10:26:00.343803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343705    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 10:26:00.343803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343762    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 10:26:00.343952 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.343839    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rvg4l\""
Sep 09 10:26:00.344283 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.344265    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 10:26:00.344283 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.344278    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:26:00.345534 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.345518    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.347767 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.347692    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.349140 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.349124    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bshz9\""
Sep 09 10:26:00.350172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.350157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:00.350242 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.350219    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:00.350292 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.350265    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 10:26:00.350327 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.350285    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 10:26:00.350327 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.350309    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 10:26:00.350923 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.350886    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vk6hs\""
Sep 09 10:26:00.351267 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.351251    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 10:26:00.351560 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.351546    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 10:26:00.352418 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.352401    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:00.353323 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353304    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4th\" (UniqueName: \"kubernetes.io/projected/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-kube-api-access-5r4th\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.353425 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353336    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysconfig\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.353425 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353361    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-systemd\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.353425 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353384    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-env-overrides\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.353425 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353409    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovnkube-script-lib\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.353617 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353434    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-system-cni-dir\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.353617 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353461    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-kubernetes\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.353617 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353484    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdmp\" (UniqueName: \"kubernetes.io/projected/eb245615-6100-4eee-9f56-75aa13793dcb-kube-api-access-cfdmp\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.353617 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb245615-6100-4eee-9f56-75aa13793dcb-tmp\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.353617 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353529    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-72tcv\""
Sep 09 10:26:00.353617 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353554    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.353617 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353591    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.353617 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353566    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rplc9\""
Sep 09 10:26:00.354010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353618    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:00.354010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353666    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysctl-d\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.354010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353691    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-kubelet\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353714    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-systemd-units\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353737    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-node-log\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.353761    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysctl-conf\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.354280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-etc-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354121    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovn-node-metrics-cert\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354154    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2028e44e-02bc-44ab-805f-135d6b34343e-dbus\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.354280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354186    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-cni-netd\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354215    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-var-lib-kubelet\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.354280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354275    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.354564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354302    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 10:26:00.354564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354353    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-lib-modules\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.354564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354377    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-slash\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354394    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blstm\" (UniqueName: \"kubernetes.io/projected/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-kube-api-access-blstm\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.354564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354439    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cnibin\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.354564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354513    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354568    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354605    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67c5131f-2142-4b96-8569-273ccb87b64e-host-slash\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354634    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb245615-6100-4eee-9f56-75aa13793dcb-etc-tuned\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354660    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2028e44e-02bc-44ab-805f-135d6b34343e-kubelet-config\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354708    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4737cd63-ad67-44bf-9164-f1a50aa8eda7-tmp-dir\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354738    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-modprobe-d\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354769    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354802    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-log-socket\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354828    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67c5131f-2142-4b96-8569-273ccb87b64e-iptables-alerter-script\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.354882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354858    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4737cd63-ad67-44bf-9164-f1a50aa8eda7-hosts-file\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354963    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-systemd\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.354985    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355002    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-serviceca\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355016    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-sys\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355034    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-host\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355072    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-var-lib-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355098    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqqb6\" (UniqueName: \"kubernetes.io/projected/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-kube-api-access-dqqb6\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355117    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d55bg\" (UniqueName: \"kubernetes.io/projected/4737cd63-ad67-44bf-9164-f1a50aa8eda7-kube-api-access-d55bg\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355132    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-run-netns\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355150    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crrdg\" (UniqueName: \"kubernetes.io/projected/67c5131f-2142-4b96-8569-273ccb87b64e-kube-api-access-crrdg\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355167    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-ovn\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355185    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-cni-bin\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355200    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovnkube-config\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355225    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-host\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.355370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355304    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-run\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.356135 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.355481    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.356135 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.356020    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 10:26:00.356352 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.356296    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 10:26:00.356434 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.356418    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9bgk6\""
Sep 09 10:26:00.358679 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.358658    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 10:26:00.358788 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.358713    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 10:26:00.358856 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.358798    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 10:26:00.358941 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.358924    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8g9hk\""
Sep 09 10:26:00.365030 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.365014    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 10:26:00.395096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.395071    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:26:00.444838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.444804    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 10:26:00.456478 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456339    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crrdg\" (UniqueName: \"kubernetes.io/projected/67c5131f-2142-4b96-8569-273ccb87b64e-kube-api-access-crrdg\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.456478 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456383    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-ovn\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.456478 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456411    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-cni-bin\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.456756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456477    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-cni-bin\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.456756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456488    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-ovn\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.456756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456638    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovnkube-config\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.456756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456671    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-host\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.456756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456698    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-socket-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.456756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456733    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-run\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456757    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4th\" (UniqueName: \"kubernetes.io/projected/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-kube-api-access-5r4th\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456785    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysconfig\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-systemd\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456832    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-env-overrides\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456857    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovnkube-script-lib\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456863    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-host\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456917    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456941    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96pz8\" (UniqueName: \"kubernetes.io/projected/8166f41c-a25f-4593-91d3-15109cdee700-kube-api-access-96pz8\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456966    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-system-cni-dir\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.456964    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysconfig\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-run\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457052    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-system-cni-dir\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457083    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-kubernetes\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457100    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-systemd\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457107    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdmp\" (UniqueName: \"kubernetes.io/projected/eb245615-6100-4eee-9f56-75aa13793dcb-kube-api-access-cfdmp\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457133    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38cd53b9-6167-4f5b-a272-baa8f3a674f3-konnectivity-ca\") pod \"konnectivity-agent-mgvw6\" (UID: \"38cd53b9-6167-4f5b-a272-baa8f3a674f3\") " pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457173    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-system-cni-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457197    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-kubernetes\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457275    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb245615-6100-4eee-9f56-75aa13793dcb-tmp\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457316    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457341    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmblv\" (UniqueName: \"kubernetes.io/projected/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-kube-api-access-kmblv\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-device-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457394    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-cni-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457405    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457416    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-kubelet\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457445    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457473    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457520    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysctl-d\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457544    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-kubelet\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457583    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-systemd-units\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457598    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.457818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457623    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-node-log\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457651    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-multus-certs\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457704    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysctl-conf\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457730    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-etc-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457775    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovn-node-metrics-cert\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457804    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2028e44e-02bc-44ab-805f-135d6b34343e-dbus\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457805    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-kubelet\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457832    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-node-log\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457867    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-registration-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457911    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-sys-fs\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457962    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-hostroot\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457972    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysctl-conf\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.457987    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-cni-netd\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458023    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458035    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-etc-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458091    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovnkube-config\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458099    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-cni-netd\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458041    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-k8s-cni-cncf-io\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.458676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458126    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-sysctl-d\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458151    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-conf-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458182    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-var-lib-kubelet\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458043    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-systemd-units\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458190    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2028e44e-02bc-44ab-805f-135d6b34343e-dbus\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458177    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovnkube-script-lib\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458229    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458295    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-var-lib-kubelet\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-lib-modules\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458347    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-slash\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458391    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-slash\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blstm\" (UniqueName: \"kubernetes.io/projected/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-kube-api-access-blstm\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458433    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458461    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-socket-dir-parent\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458486    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-lib-modules\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cnibin\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458518    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.459643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458536    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-env-overrides\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458546    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cnibin\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458545    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67c5131f-2142-4b96-8569-273ccb87b64e-host-slash\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458624    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb245615-6100-4eee-9f56-75aa13793dcb-etc-tuned\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458649    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2028e44e-02bc-44ab-805f-135d6b34343e-kubelet-config\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458674    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvfj\" (UniqueName: \"kubernetes.io/projected/15f88254-1f0c-4dc7-987a-8d0ecc54b817-kube-api-access-tdvfj\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458697    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4737cd63-ad67-44bf-9164-f1a50aa8eda7-tmp-dir\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458737    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-modprobe-d\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458758    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458777    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-log-socket\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458796    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-cnibin\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458806    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458820    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-cni-multus\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67c5131f-2142-4b96-8569-273ccb87b64e-iptables-alerter-script\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458935    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4737cd63-ad67-44bf-9164-f1a50aa8eda7-hosts-file\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458940    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.460540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458957    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-systemd\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458977    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.458980    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-log-socket\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459007    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-serviceca\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459010    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459018    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459037    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-daemon-config\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459061    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-sys\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459089    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4737cd63-ad67-44bf-9164-f1a50aa8eda7-hosts-file\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459097    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459119    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-run-systemd\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459138    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459163    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2028e44e-02bc-44ab-805f-135d6b34343e-kubelet-config\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459147    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67c5131f-2142-4b96-8569-273ccb87b64e-host-slash\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459211    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-sys\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.459276    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:00.461409 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.459342    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret podName:2028e44e-02bc-44ab-805f-135d6b34343e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:00.959323657 +0000 UTC m=+3.014882685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret") pod "global-pull-secret-syncer-4bqkk" (UID: "2028e44e-02bc-44ab-805f-135d6b34343e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459346    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4737cd63-ad67-44bf-9164-f1a50aa8eda7-tmp-dir\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459414    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-etc-modprobe-d\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459447    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-host\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459476    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-var-lib-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459496    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqqb6\" (UniqueName: \"kubernetes.io/projected/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-kube-api-access-dqqb6\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459529    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-netns\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459564    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb245615-6100-4eee-9f56-75aa13793dcb-host\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459577    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-cni-bin\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459715    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-etc-kubernetes\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459741    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-var-lib-openvswitch\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459743    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-serviceca\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459750    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55bg\" (UniqueName: \"kubernetes.io/projected/4737cd63-ad67-44bf-9164-f1a50aa8eda7-kube-api-access-d55bg\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459798    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-run-netns\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459837    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38cd53b9-6167-4f5b-a272-baa8f3a674f3-agent-certs\") pod \"konnectivity-agent-mgvw6\" (UID: \"38cd53b9-6167-4f5b-a272-baa8f3a674f3\") " pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15f88254-1f0c-4dc7-987a-8d0ecc54b817-cni-binary-copy\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.459932    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-host-run-netns\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.460265    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67c5131f-2142-4b96-8569-273ccb87b64e-iptables-alerter-script\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.462234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.460416    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r47dg"
Sep 09 10:26:00.463382 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.462878    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb245615-6100-4eee-9f56-75aa13793dcb-tmp\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.463382 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.463009    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-ovn-node-metrics-cert\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.463562 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.463525    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb245615-6100-4eee-9f56-75aa13793dcb-etc-tuned\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.473987 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.473933    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal" event={"ID":"1fb08136825eb2ae62fa73ecf8896a9e","Type":"ContainerStarted","Data":"9027bde1ed9d353d460bf927a062978f7f30b829d6c327bc02dd13a16b5d0148"}
Sep 09 10:26:00.475511 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.475480    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal" event={"ID":"b918f93a5380fe0ffc07a388ced93b00","Type":"ContainerStarted","Data":"82910571ca4367669fc3b0b00e9406245c1b82e73a1bf728b8e119b5351b1f7c"}
Sep 09 10:26:00.487714 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.487654    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r47dg"
Sep 09 10:26:00.493562 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.493542    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:00.493664 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.493567    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:00.493664 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.493583    2586 projected.go:196] Error preparing data for projected volume kube-api-access-9k5zx for pod openshift-network-diagnostics/network-check-target-x2jbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:00.493664 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.493637    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx podName:204ef70c-4a61-4e0b-a58b-d3d0e43478bc nodeName:}" failed. No retries permitted until 2026-09-09 10:26:00.993622898 +0000 UTC m=+3.049181939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9k5zx" (UniqueName: "kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx") pod "network-check-target-x2jbl" (UID: "204ef70c-4a61-4e0b-a58b-d3d0e43478bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:00.496037 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.496016    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blstm\" (UniqueName: \"kubernetes.io/projected/c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886-kube-api-access-blstm\") pod \"node-ca-dl5gz\" (UID: \"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886\") " pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.496480 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.496459    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdmp\" (UniqueName: \"kubernetes.io/projected/eb245615-6100-4eee-9f56-75aa13793dcb-kube-api-access-cfdmp\") pod \"tuned-mh9h5\" (UID: \"eb245615-6100-4eee-9f56-75aa13793dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.499141 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.499119    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crrdg\" (UniqueName: \"kubernetes.io/projected/67c5131f-2142-4b96-8569-273ccb87b64e-kube-api-access-crrdg\") pod \"iptables-alerter-sdmdp\" (UID: \"67c5131f-2142-4b96-8569-273ccb87b64e\") " pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.499339 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.499316    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4th\" (UniqueName: \"kubernetes.io/projected/0e2eed9b-16da-4a0f-a862-9cbe6792ca73-kube-api-access-5r4th\") pod \"multus-additional-cni-plugins-gjvjf\" (UID: \"0e2eed9b-16da-4a0f-a862-9cbe6792ca73\") " pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.499535 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.499511    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqqb6\" (UniqueName: \"kubernetes.io/projected/a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f-kube-api-access-dqqb6\") pod \"ovnkube-node-bhdm7\" (UID: \"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.499954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.499916    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d55bg\" (UniqueName: \"kubernetes.io/projected/4737cd63-ad67-44bf-9164-f1a50aa8eda7-kube-api-access-d55bg\") pod \"node-resolver-flzz6\" (UID: \"4737cd63-ad67-44bf-9164-f1a50aa8eda7\") " pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.560549 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560513    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-netns\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.560711 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560556    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-cni-bin\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.560711 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560573    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-etc-kubernetes\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.560711 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560593    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38cd53b9-6167-4f5b-a272-baa8f3a674f3-agent-certs\") pod \"konnectivity-agent-mgvw6\" (UID: \"38cd53b9-6167-4f5b-a272-baa8f3a674f3\") " pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:00.560711 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560639    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-cni-bin\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.560711 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560638    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-netns\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.560711 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560660    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-etc-kubernetes\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.560711 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560666    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15f88254-1f0c-4dc7-987a-8d0ecc54b817-cni-binary-copy\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560718    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-socket-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560749    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560775    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96pz8\" (UniqueName: \"kubernetes.io/projected/8166f41c-a25f-4593-91d3-15109cdee700-kube-api-access-96pz8\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560801    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38cd53b9-6167-4f5b-a272-baa8f3a674f3-konnectivity-ca\") pod \"konnectivity-agent-mgvw6\" (UID: \"38cd53b9-6167-4f5b-a272-baa8f3a674f3\") " pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560823    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-system-cni-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560850    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmblv\" (UniqueName: \"kubernetes.io/projected/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-kube-api-access-kmblv\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560866    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-socket-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560873    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-device-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560931    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-cni-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560938    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-device-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.560956    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-kubelet\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.561030    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561034    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-system-cni-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.561087    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:01.061067898 +0000 UTC m=+3.116626940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:00.561087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561094    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-cni-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561139    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-multus-certs\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561168    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-registration-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561191    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-sys-fs\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561214    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-hostroot\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561220    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15f88254-1f0c-4dc7-987a-8d0ecc54b817-cni-binary-copy\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561237    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-k8s-cni-cncf-io\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561261    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-conf-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561289    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561298    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-registration-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561326    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-socket-dir-parent\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561350    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561365    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvfj\" (UniqueName: \"kubernetes.io/projected/15f88254-1f0c-4dc7-987a-8d0ecc54b817-kube-api-access-tdvfj\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561389    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-hostroot\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561393    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-conf-dir\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561395    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-cnibin\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561431    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-k8s-cni-cncf-io\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561434    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-cni-multus\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.561773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561455    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-socket-dir-parent\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561467    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-kubelet\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561469    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-daemon-config\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561519    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561648    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561656    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38cd53b9-6167-4f5b-a272-baa8f3a674f3-konnectivity-ca\") pod \"konnectivity-agent-mgvw6\" (UID: \"38cd53b9-6167-4f5b-a272-baa8f3a674f3\") " pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561665    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-var-lib-cni-multus\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561351    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8166f41c-a25f-4593-91d3-15109cdee700-sys-fs\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561687    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-host-run-multus-certs\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561721    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15f88254-1f0c-4dc7-987a-8d0ecc54b817-cnibin\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.562346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.561862    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15f88254-1f0c-4dc7-987a-8d0ecc54b817-multus-daemon-config\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.563270 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.563235    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38cd53b9-6167-4f5b-a272-baa8f3a674f3-agent-certs\") pod \"konnectivity-agent-mgvw6\" (UID: \"38cd53b9-6167-4f5b-a272-baa8f3a674f3\") " pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:00.607810 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.607773    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmblv\" (UniqueName: \"kubernetes.io/projected/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-kube-api-access-kmblv\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:00.608300 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.608278    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96pz8\" (UniqueName: \"kubernetes.io/projected/8166f41c-a25f-4593-91d3-15109cdee700-kube-api-access-96pz8\") pod \"aws-ebs-csi-driver-node-bkc5c\" (UID: \"8166f41c-a25f-4593-91d3-15109cdee700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.608401 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.608355    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvfj\" (UniqueName: \"kubernetes.io/projected/15f88254-1f0c-4dc7-987a-8d0ecc54b817-kube-api-access-tdvfj\") pod \"multus-s5x6h\" (UID: \"15f88254-1f0c-4dc7-987a-8d0ecc54b817\") " pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.639309 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.639281    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mh9h5"
Sep 09 10:26:00.645049 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.645025    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:00.651914 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.651221    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb245615_6100_4eee_9f56_75aa13793dcb.slice/crio-e92181fc2f8e0dd8a42a64917fb7ae257a43424696a10ac76714aa184249b981 WatchSource:0}: Error finding container e92181fc2f8e0dd8a42a64917fb7ae257a43424696a10ac76714aa184249b981: Status 404 returned error can't find the container with id e92181fc2f8e0dd8a42a64917fb7ae257a43424696a10ac76714aa184249b981
Sep 09 10:26:00.652395 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.652377    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sdmdp"
Sep 09 10:26:00.657318 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.657295    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e6f7f9_612e_43ca_b787_d0e5ae6ad11f.slice/crio-9675ff2c249d3f07b4c961d949e5cbb6ce31d5a42232143f6a88811008b3dcfc WatchSource:0}: Error finding container 9675ff2c249d3f07b4c961d949e5cbb6ce31d5a42232143f6a88811008b3dcfc: Status 404 returned error can't find the container with id 9675ff2c249d3f07b4c961d949e5cbb6ce31d5a42232143f6a88811008b3dcfc
Sep 09 10:26:00.659682 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.659666    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flzz6"
Sep 09 10:26:00.661767 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.661743    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c5131f_2142_4b96_8569_273ccb87b64e.slice/crio-c7ea0e742df17ebac4693e71d0b62835c706b2fcea5b960d549665d5a9e04dd3 WatchSource:0}: Error finding container c7ea0e742df17ebac4693e71d0b62835c706b2fcea5b960d549665d5a9e04dd3: Status 404 returned error can't find the container with id c7ea0e742df17ebac4693e71d0b62835c706b2fcea5b960d549665d5a9e04dd3
Sep 09 10:26:00.664778 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.664760    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dl5gz"
Sep 09 10:26:00.670503 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.670360    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gjvjf"
Sep 09 10:26:00.675349 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.675173    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5x6h"
Sep 09 10:26:00.676379 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.676349    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4acaab0_9fe1_4c34_8bb9_45ba5d8bf886.slice/crio-faf5b4ac8a11c59ad2313222cf9c20391e24430878d49deb3bc171f7e0ead02b WatchSource:0}: Error finding container faf5b4ac8a11c59ad2313222cf9c20391e24430878d49deb3bc171f7e0ead02b: Status 404 returned error can't find the container with id faf5b4ac8a11c59ad2313222cf9c20391e24430878d49deb3bc171f7e0ead02b
Sep 09 10:26:00.680359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.680173    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:00.684471 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.684452    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2eed9b_16da_4a0f_a862_9cbe6792ca73.slice/crio-654d6734c479d8b65ecc501bb6590ea1cc79c63ec8e49526d445f72e8899e186 WatchSource:0}: Error finding container 654d6734c479d8b65ecc501bb6590ea1cc79c63ec8e49526d445f72e8899e186: Status 404 returned error can't find the container with id 654d6734c479d8b65ecc501bb6590ea1cc79c63ec8e49526d445f72e8899e186
Sep 09 10:26:00.686513 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.686494    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c"
Sep 09 10:26:00.687173 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.687150    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f88254_1f0c_4dc7_987a_8d0ecc54b817.slice/crio-33a83ce210b1a8ce6e1c2922b58cf5e79cbaf5ea5c304335e0290ca45eb7da00 WatchSource:0}: Error finding container 33a83ce210b1a8ce6e1c2922b58cf5e79cbaf5ea5c304335e0290ca45eb7da00: Status 404 returned error can't find the container with id 33a83ce210b1a8ce6e1c2922b58cf5e79cbaf5ea5c304335e0290ca45eb7da00
Sep 09 10:26:00.695218 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.695198    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cd53b9_6167_4f5b_a272_baa8f3a674f3.slice/crio-6ee92d6d30fc34eff4a003be89a3c2e37bf206f6b38b2394331c4d8399284ba6 WatchSource:0}: Error finding container 6ee92d6d30fc34eff4a003be89a3c2e37bf206f6b38b2394331c4d8399284ba6: Status 404 returned error can't find the container with id 6ee92d6d30fc34eff4a003be89a3c2e37bf206f6b38b2394331c4d8399284ba6
Sep 09 10:26:00.699385 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:00.699353    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8166f41c_a25f_4593_91d3_15109cdee700.slice/crio-fc9fb24b46ac26155a9368a51a26d5998cad93ad3dba9e48298bf5b81c612454 WatchSource:0}: Error finding container fc9fb24b46ac26155a9368a51a26d5998cad93ad3dba9e48298bf5b81c612454: Status 404 returned error can't find the container with id fc9fb24b46ac26155a9368a51a26d5998cad93ad3dba9e48298bf5b81c612454
Sep 09 10:26:00.964181 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:00.964143    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:00.964383 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.964350    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:00.964456 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:00.964429    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret podName:2028e44e-02bc-44ab-805f-135d6b34343e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:01.964407341 +0000 UTC m=+4.019966392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret") pod "global-pull-secret-syncer-4bqkk" (UID: "2028e44e-02bc-44ab-805f-135d6b34343e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:01.065086 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.065050    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:01.065272 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.065105    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:01.065272 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:01.065227    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:01.065383 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:01.065303    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:02.065284173 +0000 UTC m=+4.120843217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:01.065383 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:01.065239    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:01.065383 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:01.065341    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:01.065383 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:01.065356    2586 projected.go:196] Error preparing data for projected volume kube-api-access-9k5zx for pod openshift-network-diagnostics/network-check-target-x2jbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:01.065591 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:01.065409    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx podName:204ef70c-4a61-4e0b-a58b-d3d0e43478bc nodeName:}" failed. No retries permitted until 2026-09-09 10:26:02.065393629 +0000 UTC m=+4.120952660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k5zx" (UniqueName: "kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx") pod "network-check-target-x2jbl" (UID: "204ef70c-4a61-4e0b-a58b-d3d0e43478bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:01.480521 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.480483    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mgvw6" event={"ID":"38cd53b9-6167-4f5b-a272-baa8f3a674f3","Type":"ContainerStarted","Data":"6ee92d6d30fc34eff4a003be89a3c2e37bf206f6b38b2394331c4d8399284ba6"}
Sep 09 10:26:01.482362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.482328    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" event={"ID":"0e2eed9b-16da-4a0f-a862-9cbe6792ca73","Type":"ContainerStarted","Data":"654d6734c479d8b65ecc501bb6590ea1cc79c63ec8e49526d445f72e8899e186"}
Sep 09 10:26:01.484517 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.484492    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dl5gz" event={"ID":"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886","Type":"ContainerStarted","Data":"faf5b4ac8a11c59ad2313222cf9c20391e24430878d49deb3bc171f7e0ead02b"}
Sep 09 10:26:01.486374 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.486352    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flzz6" event={"ID":"4737cd63-ad67-44bf-9164-f1a50aa8eda7","Type":"ContainerStarted","Data":"b3701893a9f49d299199f7ebce98fa613bbe032d52d4443cca94b5e3f157d0bd"}
Sep 09 10:26:01.488059 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.488036    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sdmdp" event={"ID":"67c5131f-2142-4b96-8569-273ccb87b64e","Type":"ContainerStarted","Data":"c7ea0e742df17ebac4693e71d0b62835c706b2fcea5b960d549665d5a9e04dd3"}
Sep 09 10:26:01.490205 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.490163    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5x6h" event={"ID":"15f88254-1f0c-4dc7-987a-8d0ecc54b817","Type":"ContainerStarted","Data":"33a83ce210b1a8ce6e1c2922b58cf5e79cbaf5ea5c304335e0290ca45eb7da00"}
Sep 09 10:26:01.491659 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.491622    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 10:21:00 +0000 UTC" deadline="2028-06-06 02:12:22.452741224 +0000 UTC"
Sep 09 10:26:01.491748 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.491659    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15255h46m20.961086833s"
Sep 09 10:26:01.492537 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.492512    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"9675ff2c249d3f07b4c961d949e5cbb6ce31d5a42232143f6a88811008b3dcfc"}
Sep 09 10:26:01.495500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.495479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mh9h5" event={"ID":"eb245615-6100-4eee-9f56-75aa13793dcb","Type":"ContainerStarted","Data":"e92181fc2f8e0dd8a42a64917fb7ae257a43424696a10ac76714aa184249b981"}
Sep 09 10:26:01.496954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.496929    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c" event={"ID":"8166f41c-a25f-4593-91d3-15109cdee700","Type":"ContainerStarted","Data":"fc9fb24b46ac26155a9368a51a26d5998cad93ad3dba9e48298bf5b81c612454"}
Sep 09 10:26:01.973719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:01.973681    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:01.973937 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:01.973829    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:01.973937 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:01.973908    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret podName:2028e44e-02bc-44ab-805f-135d6b34343e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:03.973874575 +0000 UTC m=+6.029433626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret") pod "global-pull-secret-syncer-4bqkk" (UID: "2028e44e-02bc-44ab-805f-135d6b34343e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:02.074718 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:02.074683    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:02.074936 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:02.074741    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:02.075011 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.074934    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:02.075011 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.074957    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:02.075011 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.074970    2586 projected.go:196] Error preparing data for projected volume kube-api-access-9k5zx for pod openshift-network-diagnostics/network-check-target-x2jbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:02.075161 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.075034    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx podName:204ef70c-4a61-4e0b-a58b-d3d0e43478bc nodeName:}" failed. No retries permitted until 2026-09-09 10:26:04.075016799 +0000 UTC m=+6.130575831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k5zx" (UniqueName: "kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx") pod "network-check-target-x2jbl" (UID: "204ef70c-4a61-4e0b-a58b-d3d0e43478bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:02.075161 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.075107    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:02.075161 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.075142    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:04.075130499 +0000 UTC m=+6.130689544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:02.313184 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:02.313098    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 10:26:02.469071 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:02.469036    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:02.469254 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.469172    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:02.469254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:02.469036    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:02.469389 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.469273    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:02.469389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:02.469298    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:02.469389 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:02.469378    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:02.492028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:02.491968    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 10:21:00 +0000 UTC" deadline="2028-06-13 12:46:15.32609767 +0000 UTC"
Sep 09 10:26:02.492028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:02.492025    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15434h20m12.834075881s"
Sep 09 10:26:03.991965 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:03.991929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:03.996584 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:03.992069    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:03.996584 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:03.992134    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret podName:2028e44e-02bc-44ab-805f-135d6b34343e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:07.992116197 +0000 UTC m=+10.047675231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret") pod "global-pull-secret-syncer-4bqkk" (UID: "2028e44e-02bc-44ab-805f-135d6b34343e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:04.093231 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:04.093195    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:04.093231 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:04.093233    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:04.093462 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.093359    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:04.093462 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.093374    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:04.093462 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.093383    2586 projected.go:196] Error preparing data for projected volume kube-api-access-9k5zx for pod openshift-network-diagnostics/network-check-target-x2jbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:04.093462 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.093440    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx podName:204ef70c-4a61-4e0b-a58b-d3d0e43478bc nodeName:}" failed. No retries permitted until 2026-09-09 10:26:08.0934207 +0000 UTC m=+10.148979763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k5zx" (UniqueName: "kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx") pod "network-check-target-x2jbl" (UID: "204ef70c-4a61-4e0b-a58b-d3d0e43478bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:04.093834 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.093746    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:04.093834 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.093785    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:08.093774083 +0000 UTC m=+10.149333110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:04.469935 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:04.468671    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:04.469935 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.468824    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:04.469935 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:04.469119    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:04.469935 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.469214    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:04.469935 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:04.469305    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:04.469935 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:04.469467    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:06.469628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:06.469040    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:06.469628 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:06.469180    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:06.469628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:06.469258    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:06.469628 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:06.469346    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:06.469628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:06.469412    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:06.469628 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:06.469492    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:08.031967 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:08.031922    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:08.032560 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.032081    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:08.032560 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.032158    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret podName:2028e44e-02bc-44ab-805f-135d6b34343e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:16.032137829 +0000 UTC m=+18.087696884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret") pod "global-pull-secret-syncer-4bqkk" (UID: "2028e44e-02bc-44ab-805f-135d6b34343e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:08.133134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:08.133094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:08.133322 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:08.133153    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:08.133408 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.133336    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:08.133408 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.133355    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:08.133408 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.133372    2586 projected.go:196] Error preparing data for projected volume kube-api-access-9k5zx for pod openshift-network-diagnostics/network-check-target-x2jbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:08.133572 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.133433    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx podName:204ef70c-4a61-4e0b-a58b-d3d0e43478bc nodeName:}" failed. No retries permitted until 2026-09-09 10:26:16.133411834 +0000 UTC m=+18.188971060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k5zx" (UniqueName: "kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx") pod "network-check-target-x2jbl" (UID: "204ef70c-4a61-4e0b-a58b-d3d0e43478bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:08.133870 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.133775    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:08.133870 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.133852    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:16.133833947 +0000 UTC m=+18.189392998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:08.470075 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:08.469993    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:08.470075 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:08.470021    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:08.470315 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:08.470128    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:08.470315 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.470143    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:08.470315 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.470235    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:08.470458 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:08.470331    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:10.469120 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:10.469088    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:10.469564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:10.469155    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:10.469564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:10.469088    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:10.469564 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:10.469217    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:10.469564 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:10.469267    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:10.469564 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:10.469370    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:12.468316 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:12.468277    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:12.468781 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:12.468282    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:12.468781 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:12.468405    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:12.468781 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:12.468407    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:12.468781 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:12.468487    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:12.468781 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:12.468595    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:14.468428 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:14.468392    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:14.469025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:14.468436    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:14.469025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:14.468392    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:14.469025 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:14.468553    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:14.469025 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:14.468665    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:14.469025 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:14.468738    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:16.095094 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:16.095049    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:16.095535 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.095200    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:16.095535 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.095257    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret podName:2028e44e-02bc-44ab-805f-135d6b34343e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:32.095244115 +0000 UTC m=+34.150803144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret") pod "global-pull-secret-syncer-4bqkk" (UID: "2028e44e-02bc-44ab-805f-135d6b34343e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 10:26:16.195784 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:16.195751    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:16.195972 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:16.195797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:16.196027 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.195968    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:16.196027 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.195985    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 10:26:16.196027 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.195999    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 10:26:16.196027 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.196012    2586 projected.go:196] Error preparing data for projected volume kube-api-access-9k5zx for pod openshift-network-diagnostics/network-check-target-x2jbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:16.196200 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.196045    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:32.196023214 +0000 UTC m=+34.251582257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 10:26:16.196200 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.196061    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx podName:204ef70c-4a61-4e0b-a58b-d3d0e43478bc nodeName:}" failed. No retries permitted until 2026-09-09 10:26:32.196055072 +0000 UTC m=+34.251614103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k5zx" (UniqueName: "kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx") pod "network-check-target-x2jbl" (UID: "204ef70c-4a61-4e0b-a58b-d3d0e43478bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 10:26:16.468686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:16.468637    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:16.468865 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.468764    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:16.468865 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:16.468637    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:16.468865 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:16.468637    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:16.468865 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.468859    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:16.469094 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:16.468922    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:18.470661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.469438    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:18.470661 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:18.469739    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:18.470661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.470002    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:18.470661 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:18.470100    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:18.470661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.470150    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:18.470661 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:18.470228    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:18.541774 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.541731    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5x6h" event={"ID":"15f88254-1f0c-4dc7-987a-8d0ecc54b817","Type":"ContainerStarted","Data":"bf252116f5f65bea55f7bba22a2a831dd6b3dffa342ced77b28debc7689afb8b"}
Sep 09 10:26:18.545717 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.545603    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"98ce31e82eb21e9aef2879f67b78ef20be41440eac567745951adf9a0ac9615f"}
Sep 09 10:26:18.545717 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.545685    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"dde67c310f28f053d016371811967e151b765d742058cf1e7072a813a2f74709"}
Sep 09 10:26:18.547290 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.547266    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mh9h5" event={"ID":"eb245615-6100-4eee-9f56-75aa13793dcb","Type":"ContainerStarted","Data":"f1117108cbf5e11d49c5690bdb1c041c6ffaa59a8a4863e3e53f874e4994492f"}
Sep 09 10:26:18.549469 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.549446    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal" event={"ID":"b918f93a5380fe0ffc07a388ced93b00","Type":"ContainerStarted","Data":"4783bbb1d1495382f224acf5b04bcb6ec81c943de4dc88ed37d09c162fe771e6"}
Sep 09 10:26:18.566278 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.566230    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s5x6h" podStartSLOduration=2.863530914 podStartE2EDuration="20.566216324s" totalImagesPullingTime="17.70268541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:18.565761823 +0000 UTC m=+20.621320883" watchObservedRunningTime="2026-09-09 10:26:18.566216324 +0000 UTC m=+20.621775374"
Sep 09 10:26:18.583497 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.583341    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mh9h5" podStartSLOduration=3.056154121 podStartE2EDuration="20.58332747s" totalImagesPullingTime="17.527173349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:18.582448494 +0000 UTC m=+20.638007544" watchObservedRunningTime="2026-09-09 10:26:18.58332747 +0000 UTC m=+20.638886520"
Sep 09 10:26:18.594971 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:18.594925    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal" podStartSLOduration=19.594906573 podStartE2EDuration="19.594906573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:18.59482051 +0000 UTC m=+20.650379575" watchObservedRunningTime="2026-09-09 10:26:18.594906573 +0000 UTC m=+20.650465640"
Sep 09 10:26:19.552840 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.552804    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sdmdp" event={"ID":"67c5131f-2142-4b96-8569-273ccb87b64e","Type":"ContainerStarted","Data":"e3ac5e80295e44f5404d36eb9bcb7c9e2ff1aa4a40229cadcab11ae00532e9b3"}
Sep 09 10:26:19.556058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.556023    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"bab5403607e501f3e2308f80d9b62f2c4493c956800701d2551d683c4fa9bb66"}
Sep 09 10:26:19.556166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.556125    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"61ef54e5a1f86293ad5a9b5c756e8bc7fb701af2e36b2f9542243864a7c33a96"}
Sep 09 10:26:19.556229 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.556168    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"2cddd3d823627485832d52710d1cb344fa033c7647212c5ff9a5106553679e20"}
Sep 09 10:26:19.556229 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.556211    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"deb2ac45bebacc123cda4560dba55e0a355537d087fafd8d56cf64f6f4eaaaf0"}
Sep 09 10:26:19.557486 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.557463    2586 generic.go:356] "Generic (PLEG): container finished" podID="1fb08136825eb2ae62fa73ecf8896a9e" containerID="1311e1235e502e4cbeab1b1d1d3fda0d1b0952a8c56f53e0b7f0f207d31727fe" exitCode=0
Sep 09 10:26:19.557593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.557538    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal" event={"ID":"1fb08136825eb2ae62fa73ecf8896a9e","Type":"ContainerDied","Data":"1311e1235e502e4cbeab1b1d1d3fda0d1b0952a8c56f53e0b7f0f207d31727fe"}
Sep 09 10:26:19.559102 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.559081    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c" event={"ID":"8166f41c-a25f-4593-91d3-15109cdee700","Type":"ContainerStarted","Data":"b4bd6ddf4b2d85eda380553fe2d3f3024ed21ec87cb77b362ed6dc5a0e19fde2"}
Sep 09 10:26:19.560658 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.560635    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mgvw6" event={"ID":"38cd53b9-6167-4f5b-a272-baa8f3a674f3","Type":"ContainerStarted","Data":"9ae79ab48840273f68bcf11b0a7f5b4d17974a191e47c14c014883bf59f26305"}
Sep 09 10:26:19.562179 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.562158    2586 generic.go:356] "Generic (PLEG): container finished" podID="0e2eed9b-16da-4a0f-a862-9cbe6792ca73" containerID="faf4016e8d2e4d5821e45f6f949a5d65594bdcfefdbf1d9708343d5d9a514507" exitCode=0
Sep 09 10:26:19.562268 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.562240    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" event={"ID":"0e2eed9b-16da-4a0f-a862-9cbe6792ca73","Type":"ContainerDied","Data":"faf4016e8d2e4d5821e45f6f949a5d65594bdcfefdbf1d9708343d5d9a514507"}
Sep 09 10:26:19.563822 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.563801    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dl5gz" event={"ID":"c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886","Type":"ContainerStarted","Data":"124c31dd4d6e876c33f650116abcaf52068dd43de7243fe8da5ee4020814e118"}
Sep 09 10:26:19.565380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.565328    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flzz6" event={"ID":"4737cd63-ad67-44bf-9164-f1a50aa8eda7","Type":"ContainerStarted","Data":"47c3e741d61f62939ed0e9c01d16aeed2f4e021d1b776d36c9b63b9933058992"}
Sep 09 10:26:19.574590 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.574555    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sdmdp" podStartSLOduration=4.382836326 podStartE2EDuration="21.574543108s" totalImagesPullingTime="17.191706782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:19.570364321 +0000 UTC m=+21.625923383" watchObservedRunningTime="2026-09-09 10:26:19.574543108 +0000 UTC m=+21.630102159"
Sep 09 10:26:19.586691 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.586646    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dl5gz" podStartSLOduration=4.0849275800000004 podStartE2EDuration="21.586633065s" totalImagesPullingTime="17.501705485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:19.586457517 +0000 UTC m=+21.642016567" watchObservedRunningTime="2026-09-09 10:26:19.586633065 +0000 UTC m=+21.642192115"
Sep 09 10:26:19.622813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.622757    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mgvw6" podStartSLOduration=4.464637963 podStartE2EDuration="21.622738645s" totalImagesPullingTime="17.158100682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:19.622697499 +0000 UTC m=+21.678256550" watchObservedRunningTime="2026-09-09 10:26:19.622738645 +0000 UTC m=+21.678297697"
Sep 09 10:26:19.654033 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.653994    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-flzz6" podStartSLOduration=4.472717787 podStartE2EDuration="21.653979233s" totalImagesPullingTime="17.181261446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:19.653502153 +0000 UTC m=+21.709061203" watchObservedRunningTime="2026-09-09 10:26:19.653979233 +0000 UTC m=+21.709538283"
Sep 09 10:26:19.690142 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:19.689931    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 10:26:20.424233 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.424120    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T10:26:19.690141444Z","UUID":"331e6aa8-6010-4adc-9918-ded311c8d76c","Handler":null,"Name":"","Endpoint":""}
Sep 09 10:26:20.426011 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.425988    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 10:26:20.426011 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.426017    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 10:26:20.468477 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.468429    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:20.468783 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.468430    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:20.468783 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:20.468554    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:20.468783 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.468583    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:20.468783 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:20.468678    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:20.468783 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:20.468767    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:20.569474 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.569432    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal" event={"ID":"1fb08136825eb2ae62fa73ecf8896a9e","Type":"ContainerStarted","Data":"7d0fe4fd44166d045b2a181a7195997a79f6efb84ecfc7932b66520fa8d0e0cb"}
Sep 09 10:26:20.571513 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.571481    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c" event={"ID":"8166f41c-a25f-4593-91d3-15109cdee700","Type":"ContainerStarted","Data":"f0e4ecf4ac30c3435326a31dd7d6131b30e0bfa6e1fe4ffe677e88f515f021e8"}
Sep 09 10:26:20.586272 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:20.586224    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal" podStartSLOduration=21.586205611 podStartE2EDuration="21.586205611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:20.585458357 +0000 UTC m=+22.641017406" watchObservedRunningTime="2026-09-09 10:26:20.586205611 +0000 UTC m=+22.641764662"
Sep 09 10:26:21.580006 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:21.579967    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"c201446aa000b3f322acd66961ed07cbf59773d79dd4137babe95ff9f92698d5"}
Sep 09 10:26:21.582569 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:21.582540    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c" event={"ID":"8166f41c-a25f-4593-91d3-15109cdee700","Type":"ContainerStarted","Data":"5702b050a699d03415675cb4c1c4b5f6ab59e974f33fda40d0b503ac0e382e6e"}
Sep 09 10:26:21.602647 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:21.602559    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkc5c" podStartSLOduration=4.272628834 podStartE2EDuration="23.602545971s" totalImagesPullingTime="19.329917137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:21.602284421 +0000 UTC m=+23.657843473" watchObservedRunningTime="2026-09-09 10:26:21.602545971 +0000 UTC m=+23.658105021"
Sep 09 10:26:22.468827 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:22.468792    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:22.469139 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:22.468835    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:22.469139 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:22.468924    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:22.469139 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:22.468964    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:22.469139 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:22.469108    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:22.469336 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:22.469192    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:24.110510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.110366    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:24.111014 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.110993    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:24.468886 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.468864    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:24.469012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.468912    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:24.469012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.468865    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:24.469098 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:24.469028    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:24.469158 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:24.469137    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:24.469237 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:24.469213    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:24.592994 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.592951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" event={"ID":"a2e6f7f9-612e-43ca-b787-d0e5ae6ad11f","Type":"ContainerStarted","Data":"d95c6dc6152419cda1ab33a01e87803ae149c176b7c3e68ac0e5502128ea2471"}
Sep 09 10:26:24.593250 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.593231    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:24.593335 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.593299    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:24.594929 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.594902    2586 generic.go:356] "Generic (PLEG): container finished" podID="0e2eed9b-16da-4a0f-a862-9cbe6792ca73" containerID="4fc66d22323df4f65d760c27b4b6fd99b47152d5bd18fc331e5449452c8375db" exitCode=0
Sep 09 10:26:24.595035 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.594980    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" event={"ID":"0e2eed9b-16da-4a0f-a862-9cbe6792ca73","Type":"ContainerDied","Data":"4fc66d22323df4f65d760c27b4b6fd99b47152d5bd18fc331e5449452c8375db"}
Sep 09 10:26:24.608576 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.608557    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:24.620820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:24.620783    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7" podStartSLOduration=9.033609968 podStartE2EDuration="26.620771302s" totalImagesPullingTime="17.587161334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:24.620444552 +0000 UTC m=+26.676003602" watchObservedRunningTime="2026-09-09 10:26:24.620771302 +0000 UTC m=+26.676330352"
Sep 09 10:26:25.597673 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:25.597625    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:25.612809 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:25.612782    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:26:26.018207 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:26.018169    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4bqkk"]
Sep 09 10:26:26.018374 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:26.018342    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:26.018456 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:26.018436    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:26.021397 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:26.021363    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x2jbl"]
Sep 09 10:26:26.021526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:26.021456    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:26.021600 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:26.021524    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:26.024339 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:26.024315    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d48rq"]
Sep 09 10:26:26.024468 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:26.024409    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:26.024524 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:26.024505    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:26.600969 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:26.600937    2586 generic.go:356] "Generic (PLEG): container finished" podID="0e2eed9b-16da-4a0f-a862-9cbe6792ca73" containerID="ef6676d601538139f21026b7e745868cdabd0645108db323fdf10dbdc24bb7ce" exitCode=0
Sep 09 10:26:26.601513 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:26.601028    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" event={"ID":"0e2eed9b-16da-4a0f-a862-9cbe6792ca73","Type":"ContainerDied","Data":"ef6676d601538139f21026b7e745868cdabd0645108db323fdf10dbdc24bb7ce"}
Sep 09 10:26:27.468954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:27.468924    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:27.469090 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:27.469030    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:27.469090 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:27.469036    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:27.469153 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:27.469122    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:27.469183 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:27.469170    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:27.469255 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:27.469240    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:28.607166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:28.607133    2586 generic.go:356] "Generic (PLEG): container finished" podID="0e2eed9b-16da-4a0f-a862-9cbe6792ca73" containerID="4d14dcd83414898c28a9258b984022e2c323c1a178fcfe35681d557fe6871429" exitCode=0
Sep 09 10:26:28.607579 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:28.607185    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" event={"ID":"0e2eed9b-16da-4a0f-a862-9cbe6792ca73","Type":"ContainerDied","Data":"4d14dcd83414898c28a9258b984022e2c323c1a178fcfe35681d557fe6871429"}
Sep 09 10:26:29.271603 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:29.271567    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:29.271761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:29.271699    2586 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 10:26:29.272462 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:29.272440    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mgvw6"
Sep 09 10:26:29.469114 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:29.469072    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:29.469114 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:29.469114    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:29.469288 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:29.469189    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:29.469288 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:29.469200    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:26:29.469369 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:29.469282    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2jbl" podUID="204ef70c-4a61-4e0b-a58b-d3d0e43478bc"
Sep 09 10:26:29.469400 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:29.469366    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bqkk" podUID="2028e44e-02bc-44ab-805f-135d6b34343e"
Sep 09 10:26:31.241985 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.241886    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeReady"
Sep 09 10:26:31.242471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.242061    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 10:26:31.281538 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.281500    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b656c84bb-9hx2l"]
Sep 09 10:26:31.284062 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.284039    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.286859 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.286830    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 10:26:31.286995 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.286859    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 10:26:31.287248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.287229    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4xzkl\""
Sep 09 10:26:31.287628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.287573    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 10:26:31.294470 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.294445    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 10:26:31.305479 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.305441    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b656c84bb-9hx2l"]
Sep 09 10:26:31.313130 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.313105    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4c2kf"]
Sep 09 10:26:31.317091 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.317066    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.317790 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.317768    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zznrb"]
Sep 09 10:26:31.321356 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.321336    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zznrb"
Sep 09 10:26:31.321554 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.321391    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-272r8\""
Sep 09 10:26:31.321622 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.321553    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 10:26:31.321705 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.321692    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 10:26:31.324004 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.323886    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wx2nx\""
Sep 09 10:26:31.325586 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.325564    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 10:26:31.326325 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.326304    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 10:26:31.326876 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.326858    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 10:26:31.333970 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.333948    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4c2kf"]
Sep 09 10:26:31.336680 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.336659    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zznrb"]
Sep 09 10:26:31.410099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410062    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-tmp-dir\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.410272 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410151    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdc76\" (UniqueName: \"kubernetes.io/projected/ad647ae4-c0ad-4e78-a358-cbfb757329f0-kube-api-access-xdc76\") pod \"ingress-canary-zznrb\" (UID: \"ad647ae4-c0ad-4e78-a358-cbfb757329f0\") " pod="openshift-ingress-canary/ingress-canary-zznrb"
Sep 09 10:26:31.410272 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410198    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-bound-sa-token\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.410272 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410223    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-config-volume\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.410434 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410251    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7t9\" (UniqueName: \"kubernetes.io/projected/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-kube-api-access-2t7t9\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.410434 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410326    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.410434 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410408    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.410557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410447    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-installation-pull-secrets\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.410557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410474    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5fpq\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-kube-api-access-l5fpq\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.410557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410542    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-image-registry-private-configuration\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.410686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410571    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad647ae4-c0ad-4e78-a358-cbfb757329f0-cert\") pod \"ingress-canary-zznrb\" (UID: \"ad647ae4-c0ad-4e78-a358-cbfb757329f0\") " pod="openshift-ingress-canary/ingress-canary-zznrb"
Sep 09 10:26:31.410686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410602    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/775fc55b-45cc-4506-b06e-822381628930-ca-trust-extracted\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.410686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410637    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-trusted-ca\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.410813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.410691    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-registry-certificates\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.468408 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.468377    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:31.468408 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.468410    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:31.468599 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.468377    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:31.471434 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.471413    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 10:26:31.471434 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.471413    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 10:26:31.471591 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.471448    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffrc8\""
Sep 09 10:26:31.472135 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.472109    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 10:26:31.472281 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.472267    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hq5pc\""
Sep 09 10:26:31.472582 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.472567    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 10:26:31.511592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.511592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511559    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-installation-pull-secrets\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.511592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511578    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5fpq\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-kube-api-access-l5fpq\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511613    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-image-registry-private-configuration\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511628    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad647ae4-c0ad-4e78-a358-cbfb757329f0-cert\") pod \"ingress-canary-zznrb\" (UID: \"ad647ae4-c0ad-4e78-a358-cbfb757329f0\") " pod="openshift-ingress-canary/ingress-canary-zznrb"
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511653    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/775fc55b-45cc-4506-b06e-822381628930-ca-trust-extracted\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511680    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-trusted-ca\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:31.511685    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:31.511706    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-registry-certificates\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511748    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-tmp-dir\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.511813 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:31.511771    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:32.011752522 +0000 UTC m=+34.067311564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511822    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdc76\" (UniqueName: \"kubernetes.io/projected/ad647ae4-c0ad-4e78-a358-cbfb757329f0-kube-api-access-xdc76\") pod \"ingress-canary-zznrb\" (UID: \"ad647ae4-c0ad-4e78-a358-cbfb757329f0\") " pod="openshift-ingress-canary/ingress-canary-zznrb"
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511853    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-bound-sa-token\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-config-volume\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511920    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7t9\" (UniqueName: \"kubernetes.io/projected/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-kube-api-access-2t7t9\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.511942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.512026    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-tmp-dir\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:31.512040    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:31.512091    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:32.012076918 +0000 UTC m=+34.067635946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:31.512306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.512168    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/775fc55b-45cc-4506-b06e-822381628930-ca-trust-extracted\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.512805 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.512784    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-config-volume\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.513096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.512901    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-trusted-ca\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.513096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.512915    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-registry-certificates\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.516436 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.516413    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-installation-pull-secrets\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.516552 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.516439    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad647ae4-c0ad-4e78-a358-cbfb757329f0-cert\") pod \"ingress-canary-zznrb\" (UID: \"ad647ae4-c0ad-4e78-a358-cbfb757329f0\") " pod="openshift-ingress-canary/ingress-canary-zznrb"
Sep 09 10:26:31.516552 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.516461    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-image-registry-private-configuration\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.524704 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.524678    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7t9\" (UniqueName: \"kubernetes.io/projected/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-kube-api-access-2t7t9\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:31.524845 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.524810    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdc76\" (UniqueName: \"kubernetes.io/projected/ad647ae4-c0ad-4e78-a358-cbfb757329f0-kube-api-access-xdc76\") pod \"ingress-canary-zznrb\" (UID: \"ad647ae4-c0ad-4e78-a358-cbfb757329f0\") " pod="openshift-ingress-canary/ingress-canary-zznrb"
Sep 09 10:26:31.525497 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.525478    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5fpq\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-kube-api-access-l5fpq\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.525571 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.525527    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-bound-sa-token\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:31.634278 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.634248    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zznrb"
Sep 09 10:26:31.932095 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:31.931854    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zznrb"]
Sep 09 10:26:32.017560 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.017532    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:32.017744 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.017568    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:32.017744 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:32.017709    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:26:32.017744 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:32.017724    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:26:32.017920 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:32.017774    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:33.017756477 +0000 UTC m=+35.073315505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:26:32.017920 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:32.017710    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:32.017920 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:32.017859    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:33.017837826 +0000 UTC m=+35.073396869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:32.118199 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.118157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:32.121132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.121105    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2028e44e-02bc-44ab-805f-135d6b34343e-original-pull-secret\") pod \"global-pull-secret-syncer-4bqkk\" (UID: \"2028e44e-02bc-44ab-805f-135d6b34343e\") " pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:32.219485 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.219395    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:26:32.219485 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.219439    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:32.219716 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:32.219589    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 10:26:32.219716 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:32.219657    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:04.2196411 +0000 UTC m=+66.275200147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : secret "metrics-daemon-secret" not found
Sep 09 10:26:32.222044 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.222022    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k5zx\" (UniqueName: \"kubernetes.io/projected/204ef70c-4a61-4e0b-a58b-d3d0e43478bc-kube-api-access-9k5zx\") pod \"network-check-target-x2jbl\" (UID: \"204ef70c-4a61-4e0b-a58b-d3d0e43478bc\") " pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:32.378081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.378035    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:32.383645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:32.383621    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bqkk"
Sep 09 10:26:33.024312 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:33.024276    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:33.024506 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:33.024321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:33.024506 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:33.024450    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:33.024506 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:33.024475    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:26:33.024506 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:33.024489    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:26:33.024662 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:33.024531    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:35.02451508 +0000 UTC m=+37.080074113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:33.024662 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:33.024552    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:35.024542515 +0000 UTC m=+37.080101551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:26:33.315237 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:33.315159    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad647ae4_c0ad_4e78_a358_cbfb757329f0.slice/crio-a288dcd66c2649517c21ee2744788bd7c4ebc93cb23ac073098659f698499d72 WatchSource:0}: Error finding container a288dcd66c2649517c21ee2744788bd7c4ebc93cb23ac073098659f698499d72: Status 404 returned error can't find the container with id a288dcd66c2649517c21ee2744788bd7c4ebc93cb23ac073098659f698499d72
Sep 09 10:26:33.590691 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:33.590667    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x2jbl"]
Sep 09 10:26:33.593090 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:33.593035    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4bqkk"]
Sep 09 10:26:33.620314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:33.620281    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zznrb" event={"ID":"ad647ae4-c0ad-4e78-a358-cbfb757329f0","Type":"ContainerStarted","Data":"a288dcd66c2649517c21ee2744788bd7c4ebc93cb23ac073098659f698499d72"}
Sep 09 10:26:33.696075 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:33.696042    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204ef70c_4a61_4e0b_a58b_d3d0e43478bc.slice/crio-b2fd071dd7f2e897821633da9f6a3b1c6d025ca3080142ef6941adbbea8aa5a0 WatchSource:0}: Error finding container b2fd071dd7f2e897821633da9f6a3b1c6d025ca3080142ef6941adbbea8aa5a0: Status 404 returned error can't find the container with id b2fd071dd7f2e897821633da9f6a3b1c6d025ca3080142ef6941adbbea8aa5a0
Sep 09 10:26:33.696824 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:26:33.696797    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2028e44e_02bc_44ab_805f_135d6b34343e.slice/crio-5cfe7698a19f53f7af34978317982576685668582af65c3ea61642377533d984 WatchSource:0}: Error finding container 5cfe7698a19f53f7af34978317982576685668582af65c3ea61642377533d984: Status 404 returned error can't find the container with id 5cfe7698a19f53f7af34978317982576685668582af65c3ea61642377533d984
Sep 09 10:26:34.627601 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:34.627425    2586 generic.go:356] "Generic (PLEG): container finished" podID="0e2eed9b-16da-4a0f-a862-9cbe6792ca73" containerID="c4746ef23da40e65bcd3df11a904902d9aabe4db1868174846089f7743a14266" exitCode=0
Sep 09 10:26:34.627601 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:34.627530    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" event={"ID":"0e2eed9b-16da-4a0f-a862-9cbe6792ca73","Type":"ContainerDied","Data":"c4746ef23da40e65bcd3df11a904902d9aabe4db1868174846089f7743a14266"}
Sep 09 10:26:34.629556 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:34.629486    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x2jbl" event={"ID":"204ef70c-4a61-4e0b-a58b-d3d0e43478bc","Type":"ContainerStarted","Data":"b2fd071dd7f2e897821633da9f6a3b1c6d025ca3080142ef6941adbbea8aa5a0"}
Sep 09 10:26:34.631482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:34.631430    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4bqkk" event={"ID":"2028e44e-02bc-44ab-805f-135d6b34343e","Type":"ContainerStarted","Data":"5cfe7698a19f53f7af34978317982576685668582af65c3ea61642377533d984"}
Sep 09 10:26:35.041717 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:35.041397    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:35.041717 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:35.041675    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:35.041717 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:35.041608    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:35.042003 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:35.041785    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:39.04176104 +0000 UTC m=+41.097320086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:35.042003 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:35.041808    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:26:35.042003 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:35.041826    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:26:35.042003 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:35.041879    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:39.041861945 +0000 UTC m=+41.097421003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:26:35.640701 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:35.640665    2586 generic.go:356] "Generic (PLEG): container finished" podID="0e2eed9b-16da-4a0f-a862-9cbe6792ca73" containerID="7b96af729f8c177e521e1a494ea8a204f68504ec3ef40c8f85f34fc0b99172b1" exitCode=0
Sep 09 10:26:35.641156 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:35.640717    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" event={"ID":"0e2eed9b-16da-4a0f-a862-9cbe6792ca73","Type":"ContainerDied","Data":"7b96af729f8c177e521e1a494ea8a204f68504ec3ef40c8f85f34fc0b99172b1"}
Sep 09 10:26:36.648264 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:36.648226    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" event={"ID":"0e2eed9b-16da-4a0f-a862-9cbe6792ca73","Type":"ContainerStarted","Data":"1d9f887f337d10b52e1124d79d952b202454819f8404f9bdf096a07194b6443c"}
Sep 09 10:26:36.672464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:36.672399    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gjvjf" podStartSLOduration=8.923621967999999 podStartE2EDuration="38.672377072s" totalImagesPullingTime="29.748755104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:36.671754259 +0000 UTC m=+38.727313341" watchObservedRunningTime="2026-09-09 10:26:36.672377072 +0000 UTC m=+38.727936125"
Sep 09 10:26:39.071696 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.071658    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:39.071696 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.071706    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:39.072195 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:39.071820    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:26:39.072195 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:39.071822    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:39.072195 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:39.071885    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:47.071869807 +0000 UTC m=+49.127428835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:39.072195 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:39.071832    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:26:39.072195 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:39.071944    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:47.071934033 +0000 UTC m=+49.127493064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:26:39.658533 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.657814    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zznrb" event={"ID":"ad647ae4-c0ad-4e78-a358-cbfb757329f0","Type":"ContainerStarted","Data":"e1badfab45cf086e47b021bf3f8318c912a4e0cbc43d25893631fb209d080f19"}
Sep 09 10:26:39.660315 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.660286    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x2jbl" event={"ID":"204ef70c-4a61-4e0b-a58b-d3d0e43478bc","Type":"ContainerStarted","Data":"1096fe324007aa2a5269bd96fc5010ca4f2274c8751056ab699cb777f55c7e3c"}
Sep 09 10:26:39.660459 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.660413    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:26:39.662052 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.662029    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4bqkk" event={"ID":"2028e44e-02bc-44ab-805f-135d6b34343e","Type":"ContainerStarted","Data":"e06a0cd5dc8dc4ad76f1df14d7b2d76293673c0559e2dac1fd9cab7beda0d5e4"}
Sep 09 10:26:39.676450 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.676400    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zznrb" podStartSLOduration=2.617796137 podStartE2EDuration="8.676381801s" totalImagesPullingTime="6.058585664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:39.675193582 +0000 UTC m=+41.730752632" watchObservedRunningTime="2026-09-09 10:26:39.676381801 +0000 UTC m=+41.731940851"
Sep 09 10:26:39.693287 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.693230    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x2jbl" podStartSLOduration=36.034669908 podStartE2EDuration="41.693211186s" totalImagesPullingTime="5.658541278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:39.692465693 +0000 UTC m=+41.748024744" watchObservedRunningTime="2026-09-09 10:26:39.693211186 +0000 UTC m=+41.748770235"
Sep 09 10:26:39.708949 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:39.708908    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4bqkk" podStartSLOduration=36.040002675 podStartE2EDuration="41.708879464s" totalImagesPullingTime="5.668876789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:39.7085054 +0000 UTC m=+41.764064452" watchObservedRunningTime="2026-09-09 10:26:39.708879464 +0000 UTC m=+41.764438514"
Sep 09 10:26:47.129079 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:47.129042    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:26:47.129079 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:47.129086    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:26:47.129517 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:47.129183    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:26:47.129517 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:47.129195    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:26:47.129517 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:47.129198    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:26:47.129517 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:47.129252    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:03.129237408 +0000 UTC m=+65.184796437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:26:47.129517 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:26:47.129266    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:03.129260058 +0000 UTC m=+65.184819086 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:26:57.613060 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:26:57.613026    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhdm7"
Sep 09 10:27:03.139628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:27:03.139583    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:27:03.139628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:27:03.139631    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:27:03.140124 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:03.139739    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:27:03.140124 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:03.139746    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:27:03.140124 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:03.139812    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:35.139795411 +0000 UTC m=+97.195354439 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:27:03.140124 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:03.139751    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:27:03.140124 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:03.139879    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:35.139867944 +0000 UTC m=+97.195426972 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:27:04.249695 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:27:04.249658    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:27:04.250099 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:04.249804    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 10:27:04.250099 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:04.249870    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:28:08.249854268 +0000 UTC m=+130.305413296 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : secret "metrics-daemon-secret" not found
Sep 09 10:27:10.667274 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:27:10.667243    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x2jbl"
Sep 09 10:27:35.169203 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:27:35.169059    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:27:35.169203 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:27:35.169139    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:27:35.169702 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:35.169233    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:27:35.169702 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:35.169238    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:27:35.169702 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:35.169260    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:27:35.169702 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:35.169296    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:28:39.16927904 +0000 UTC m=+161.224838084 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:27:35.169702 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:27:35.169317    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:28:39.16930265 +0000 UTC m=+161.224861692 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:28:08.298171 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:28:08.298130    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:28:08.298654 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:08.298255    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 10:28:08.298654 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:08.298308    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:10.298294442 +0000 UTC m=+252.353853470 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : secret "metrics-daemon-secret" not found
Sep 09 10:28:34.295517 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:34.295453    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" podUID="775fc55b-45cc-4506-b06e-822381628930"
Sep 09 10:28:34.328296 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:34.328258    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4c2kf" podUID="4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226"
Sep 09 10:28:34.487908 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:34.487862    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:28:34.937672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:28:34.937646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:28:34.937839 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:28:34.937646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:28:39.211121 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:28:39.211084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:28:39.211510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:28:39.211128    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:28:39.211510 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:39.211225    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 10:28:39.211510 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:39.211233    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 10:28:39.211510 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:39.211301    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls podName:4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:41.211286294 +0000 UTC m=+283.266845322 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls") pod "dns-default-4c2kf" (UID: "4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226") : secret "dns-default-metrics-tls" not found
Sep 09 10:28:39.211510 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:39.211238    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b656c84bb-9hx2l: secret "image-registry-tls" not found
Sep 09 10:28:39.211510 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:28:39.211375    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls podName:775fc55b-45cc-4506-b06e-822381628930 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:41.211362631 +0000 UTC m=+283.266921664 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls") pod "image-registry-5b656c84bb-9hx2l" (UID: "775fc55b-45cc-4506-b06e-822381628930") : secret "image-registry-tls" not found
Sep 09 10:28:46.468379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:28:46.468343    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:30:06.963815 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.963778    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w"]
Sep 09 10:30:06.966548 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.966532    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w"
Sep 09 10:30:06.968855 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.968826    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:30:06.968998 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.968825    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 10:30:06.968998 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.968831    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q8h6r\""
Sep 09 10:30:06.973327 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.973182    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"]
Sep 09 10:30:06.975806 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.975784    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w"]
Sep 09 10:30:06.975920 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.975864    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:06.979481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.979363    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:30:06.979579 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.979530    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qtzmn\""
Sep 09 10:30:06.979707 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.979687    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 10:30:06.980068 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.980051    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 10:30:06.980161 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.980067    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 10:30:06.996564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:06.996540    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"]
Sep 09 10:30:07.057738 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.057708    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jpm9t"]
Sep 09 10:30:07.060524 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.060508    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:07.062917 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.062878    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"]
Sep 09 10:30:07.063034 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.062919    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 10:30:07.063302 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.063283    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kp6qz\""
Sep 09 10:30:07.065715 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.065700    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-hggvk"]
Sep 09 10:30:07.065821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.065804    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:07.066308 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.066287    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 10:30:07.068299 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.068282    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 10:30:07.068407 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.068362    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 10:30:07.068469 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.068408    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qlvsm\""
Sep 09 10:30:07.068526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.068492    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:30:07.068585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.068524    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"]
Sep 09 10:30:07.068635 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.068615    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-hggvk"
Sep 09 10:30:07.071262 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.071243    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.072534 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.072511    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jpm9t"]
Sep 09 10:30:07.072691 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.072520    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-smvz5\""
Sep 09 10:30:07.073145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.073127    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 10:30:07.073924 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.073885    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 10:30:07.074756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.074684    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 10:30:07.074848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.074695    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-krd24\""
Sep 09 10:30:07.076509 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.076488    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 10:30:07.084679 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.084658    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-hggvk"]
Sep 09 10:30:07.085508 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.085490    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"]
Sep 09 10:30:07.088207 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.088185    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"]
Sep 09 10:30:07.114528 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.114500    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/572133a3-bac1-458a-b2ed-fb140a33bbe3-serving-cert\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.114667 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.114538    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8znkn\" (UniqueName: \"kubernetes.io/projected/018e1efe-ab8e-4a08-a366-56aab8b91458-kube-api-access-8znkn\") pod \"volume-data-source-validator-695c895bd6-cwb4w\" (UID: \"018e1efe-ab8e-4a08-a366-56aab8b91458\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w"
Sep 09 10:30:07.114667 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.114577    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/572133a3-bac1-458a-b2ed-fb140a33bbe3-config\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.114770 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.114666    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j74g\" (UniqueName: \"kubernetes.io/projected/572133a3-bac1-458a-b2ed-fb140a33bbe3-kube-api-access-5j74g\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.160692 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.160664    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-rmszz"]
Sep 09 10:30:07.163633 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.163618    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.168931 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.168905    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 10:30:07.169096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.168934    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 10:30:07.169096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.168979    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 10:30:07.169096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.169075    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 10:30:07.169340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.169233    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6g6vs\""
Sep 09 10:30:07.177752 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.177730    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-rmszz"]
Sep 09 10:30:07.180124 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.180105    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 10:30:07.215489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215401    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/572133a3-bac1-458a-b2ed-fb140a33bbe3-serving-cert\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.215489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215436    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8znkn\" (UniqueName: \"kubernetes.io/projected/018e1efe-ab8e-4a08-a366-56aab8b91458-kube-api-access-8znkn\") pod \"volume-data-source-validator-695c895bd6-cwb4w\" (UID: \"018e1efe-ab8e-4a08-a366-56aab8b91458\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w"
Sep 09 10:30:07.215489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215461    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd142694-acce-48c2-ba76-c2361de63250-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.215719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215491    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/572133a3-bac1-458a-b2ed-fb140a33bbe3-config\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.215719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215528    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.215719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215555    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:07.215719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215602    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j74g\" (UniqueName: \"kubernetes.io/projected/572133a3-bac1-458a-b2ed-fb140a33bbe3-kube-api-access-5j74g\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.215719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215660    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79m2\" (UniqueName: \"kubernetes.io/projected/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-kube-api-access-m79m2\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:07.215719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215714    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4f9527ed-c3ba-45eb-a51c-80922f29870a-nginx-conf\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:07.215966 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215751    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7r7x\" (UniqueName: \"kubernetes.io/projected/fd142694-acce-48c2-ba76-c2361de63250-kube-api-access-n7r7x\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.215966 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215780    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:07.215966 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.215806    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bt9\" (UniqueName: \"kubernetes.io/projected/ee452166-2437-4abb-b4ef-6345e2ce0d39-kube-api-access-87bt9\") pod \"network-check-source-685456655f-hggvk\" (UID: \"ee452166-2437-4abb-b4ef-6345e2ce0d39\") " pod="openshift-network-diagnostics/network-check-source-685456655f-hggvk"
Sep 09 10:30:07.216121 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.216104    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/572133a3-bac1-458a-b2ed-fb140a33bbe3-config\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.218524 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.218503    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/572133a3-bac1-458a-b2ed-fb140a33bbe3-serving-cert\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.229619 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.229591    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j74g\" (UniqueName: \"kubernetes.io/projected/572133a3-bac1-458a-b2ed-fb140a33bbe3-kube-api-access-5j74g\") pod \"service-ca-operator-859f94c4b-7bkz6\" (UID: \"572133a3-bac1-458a-b2ed-fb140a33bbe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.231257 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.231239    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8znkn\" (UniqueName: \"kubernetes.io/projected/018e1efe-ab8e-4a08-a366-56aab8b91458-kube-api-access-8znkn\") pod \"volume-data-source-validator-695c895bd6-cwb4w\" (UID: \"018e1efe-ab8e-4a08-a366-56aab8b91458\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w"
Sep 09 10:30:07.275725 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.275691    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w"
Sep 09 10:30:07.284939 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.284916    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"
Sep 09 10:30:07.316448 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316414    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4f9527ed-c3ba-45eb-a51c-80922f29870a-nginx-conf\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:07.316448 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79m2\" (UniqueName: \"kubernetes.io/projected/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-kube-api-access-m79m2\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:07.316655 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7r7x\" (UniqueName: \"kubernetes.io/projected/fd142694-acce-48c2-ba76-c2361de63250-kube-api-access-n7r7x\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.316655 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316491    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:07.316655 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316518    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db6c7c40-96e8-473e-ae77-35168a674dbe-serving-cert\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.316655 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316547    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db6c7c40-96e8-473e-ae77-35168a674dbe-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.316655 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316573    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsfpg\" (UniqueName: \"kubernetes.io/projected/db6c7c40-96e8-473e-ae77-35168a674dbe-kube-api-access-qsfpg\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.316655 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316590    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bt9\" (UniqueName: \"kubernetes.io/projected/ee452166-2437-4abb-b4ef-6345e2ce0d39-kube-api-access-87bt9\") pod \"network-check-source-685456655f-hggvk\" (UID: \"ee452166-2437-4abb-b4ef-6345e2ce0d39\") " pod="openshift-network-diagnostics/network-check-source-685456655f-hggvk"
Sep 09 10:30:07.316963 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.316706    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 10:30:07.316963 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.316784    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert podName:4f9527ed-c3ba-45eb-a51c-80922f29870a nodeName:}" failed. No retries permitted until 2026-09-09 10:30:07.816760779 +0000 UTC m=+249.872319819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jpm9t" (UID: "4f9527ed-c3ba-45eb-a51c-80922f29870a") : secret "networking-console-plugin-cert" not found
Sep 09 10:30:07.316963 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.316953    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db6c7c40-96e8-473e-ae77-35168a674dbe-tmp\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.317191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.317020    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd142694-acce-48c2-ba76-c2361de63250-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.317191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.317049    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:07.317191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.317081    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/db6c7c40-96e8-473e-ae77-35168a674dbe-snapshots\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.317191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.317109    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db6c7c40-96e8-473e-ae77-35168a674dbe-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.317191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.317136    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.317414 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.317260    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:07.317414 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.317318    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 10:30:07.317414 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.317353    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls podName:fd142694-acce-48c2-ba76-c2361de63250 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:07.817331397 +0000 UTC m=+249.872890448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lm2ph" (UID: "fd142694-acce-48c2-ba76-c2361de63250") : secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:07.317414 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.317404    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls podName:a99b830c-9630-4af7-8f8c-1dbd9cb7c076 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:07.817393734 +0000 UTC m=+249.872952779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gqj4p" (UID: "a99b830c-9630-4af7-8f8c-1dbd9cb7c076") : secret "samples-operator-tls" not found
Sep 09 10:30:07.317554 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.317474    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4f9527ed-c3ba-45eb-a51c-80922f29870a-nginx-conf\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:07.317816 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.317789    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd142694-acce-48c2-ba76-c2361de63250-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.327402 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.327128    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bt9\" (UniqueName: \"kubernetes.io/projected/ee452166-2437-4abb-b4ef-6345e2ce0d39-kube-api-access-87bt9\") pod \"network-check-source-685456655f-hggvk\" (UID: \"ee452166-2437-4abb-b4ef-6345e2ce0d39\") " pod="openshift-network-diagnostics/network-check-source-685456655f-hggvk"
Sep 09 10:30:07.327402 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.327318    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7r7x\" (UniqueName: \"kubernetes.io/projected/fd142694-acce-48c2-ba76-c2361de63250-kube-api-access-n7r7x\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.330174 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.329652    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79m2\" (UniqueName: \"kubernetes.io/projected/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-kube-api-access-m79m2\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:07.387401 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.387366    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-hggvk"
Sep 09 10:30:07.417641 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.417610    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db6c7c40-96e8-473e-ae77-35168a674dbe-serving-cert\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.417798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.417655    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db6c7c40-96e8-473e-ae77-35168a674dbe-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.417798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.417679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsfpg\" (UniqueName: \"kubernetes.io/projected/db6c7c40-96e8-473e-ae77-35168a674dbe-kube-api-access-qsfpg\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.417798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.417701    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db6c7c40-96e8-473e-ae77-35168a674dbe-tmp\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.417798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.417739    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/db6c7c40-96e8-473e-ae77-35168a674dbe-snapshots\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.417798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.417756    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db6c7c40-96e8-473e-ae77-35168a674dbe-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.418199 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.418169    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db6c7c40-96e8-473e-ae77-35168a674dbe-tmp\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.418488 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.418466    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db6c7c40-96e8-473e-ae77-35168a674dbe-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.418570 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.418543    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/db6c7c40-96e8-473e-ae77-35168a674dbe-snapshots\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.418848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.418829    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db6c7c40-96e8-473e-ae77-35168a674dbe-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.419967 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.419950    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db6c7c40-96e8-473e-ae77-35168a674dbe-serving-cert\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.425590 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.425564    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsfpg\" (UniqueName: \"kubernetes.io/projected/db6c7c40-96e8-473e-ae77-35168a674dbe-kube-api-access-qsfpg\") pod \"insights-operator-76bcd6ccf5-rmszz\" (UID: \"db6c7c40-96e8-473e-ae77-35168a674dbe\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.472353 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.472278    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz"
Sep 09 10:30:07.544927 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.544879    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w"]
Sep 09 10:30:07.547574 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:07.547551    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018e1efe_ab8e_4a08_a366_56aab8b91458.slice/crio-dee74912de72494b06264494633fe868001611150d1e87de42d336069883486a WatchSource:0}: Error finding container dee74912de72494b06264494633fe868001611150d1e87de42d336069883486a: Status 404 returned error can't find the container with id dee74912de72494b06264494633fe868001611150d1e87de42d336069883486a
Sep 09 10:30:07.575196 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.575161    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6"]
Sep 09 10:30:07.577767 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:07.577739    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572133a3_bac1_458a_b2ed_fb140a33bbe3.slice/crio-08edaf6987fa41f5a256d8ed6c7839070677a72fbaddcf87db0def4b83d2ecb8 WatchSource:0}: Error finding container 08edaf6987fa41f5a256d8ed6c7839070677a72fbaddcf87db0def4b83d2ecb8: Status 404 returned error can't find the container with id 08edaf6987fa41f5a256d8ed6c7839070677a72fbaddcf87db0def4b83d2ecb8
Sep 09 10:30:07.658618 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.658562    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-hggvk"]
Sep 09 10:30:07.660953 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:07.660923    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee452166_2437_4abb_b4ef_6345e2ce0d39.slice/crio-5229e3d968643724bb0d1b7042148f25c97f333e484baeeb544f829506751f9a WatchSource:0}: Error finding container 5229e3d968643724bb0d1b7042148f25c97f333e484baeeb544f829506751f9a: Status 404 returned error can't find the container with id 5229e3d968643724bb0d1b7042148f25c97f333e484baeeb544f829506751f9a
Sep 09 10:30:07.740960 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.740871    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-rmszz"]
Sep 09 10:30:07.743971 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:07.743943    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6c7c40_96e8_473e_ae77_35168a674dbe.slice/crio-0e87e8059c532820e02abf4c840d3138785ebeab8b6a7eaa6a4fea93c4560e49 WatchSource:0}: Error finding container 0e87e8059c532820e02abf4c840d3138785ebeab8b6a7eaa6a4fea93c4560e49: Status 404 returned error can't find the container with id 0e87e8059c532820e02abf4c840d3138785ebeab8b6a7eaa6a4fea93c4560e49
Sep 09 10:30:07.821312 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.821272    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:07.821495 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.821324    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:07.821495 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:07.821369    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:07.821495 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.821446    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 10:30:07.821657 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.821497    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:07.821657 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.821518    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls podName:a99b830c-9630-4af7-8f8c-1dbd9cb7c076 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:08.821497925 +0000 UTC m=+250.877056956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gqj4p" (UID: "a99b830c-9630-4af7-8f8c-1dbd9cb7c076") : secret "samples-operator-tls" not found
Sep 09 10:30:07.821657 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.821556    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls podName:fd142694-acce-48c2-ba76-c2361de63250 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:08.821540067 +0000 UTC m=+250.877099112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lm2ph" (UID: "fd142694-acce-48c2-ba76-c2361de63250") : secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:07.821657 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.821497    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 10:30:07.821657 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:07.821625    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert podName:4f9527ed-c3ba-45eb-a51c-80922f29870a nodeName:}" failed. No retries permitted until 2026-09-09 10:30:08.821606836 +0000 UTC m=+250.877165880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jpm9t" (UID: "4f9527ed-c3ba-45eb-a51c-80922f29870a") : secret "networking-console-plugin-cert" not found
Sep 09 10:30:08.164456 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.164423    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz" event={"ID":"db6c7c40-96e8-473e-ae77-35168a674dbe","Type":"ContainerStarted","Data":"0e87e8059c532820e02abf4c840d3138785ebeab8b6a7eaa6a4fea93c4560e49"}
Sep 09 10:30:08.166079 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.166037    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-hggvk" event={"ID":"ee452166-2437-4abb-b4ef-6345e2ce0d39","Type":"ContainerStarted","Data":"398cc6fe304dcb438ecc406132a7985e56d1e86aa37dad504df6b2eae92fbd4f"}
Sep 09 10:30:08.166207 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.166090    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-hggvk" event={"ID":"ee452166-2437-4abb-b4ef-6345e2ce0d39","Type":"ContainerStarted","Data":"5229e3d968643724bb0d1b7042148f25c97f333e484baeeb544f829506751f9a"}
Sep 09 10:30:08.167177 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.167148    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6" event={"ID":"572133a3-bac1-458a-b2ed-fb140a33bbe3","Type":"ContainerStarted","Data":"08edaf6987fa41f5a256d8ed6c7839070677a72fbaddcf87db0def4b83d2ecb8"}
Sep 09 10:30:08.168168 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.168143    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w" event={"ID":"018e1efe-ab8e-4a08-a366-56aab8b91458","Type":"ContainerStarted","Data":"dee74912de72494b06264494633fe868001611150d1e87de42d336069883486a"}
Sep 09 10:30:08.181406 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.181348    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-hggvk" podStartSLOduration=1.181332232 podStartE2EDuration="1.181332232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:08.180809636 +0000 UTC m=+250.236368698" watchObservedRunningTime="2026-09-09 10:30:08.181332232 +0000 UTC m=+250.236891283"
Sep 09 10:30:08.831731 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.831703    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:08.831821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.831759    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:08.831821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:08.831816    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:08.831945 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:08.831867    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 10:30:08.832003 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:08.831945    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls podName:a99b830c-9630-4af7-8f8c-1dbd9cb7c076 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:10.831925838 +0000 UTC m=+252.887484875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gqj4p" (UID: "a99b830c-9630-4af7-8f8c-1dbd9cb7c076") : secret "samples-operator-tls" not found
Sep 09 10:30:08.832003 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:08.831953    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:08.832112 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:08.832022    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls podName:fd142694-acce-48c2-ba76-c2361de63250 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:10.83200401 +0000 UTC m=+252.887563057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lm2ph" (UID: "fd142694-acce-48c2-ba76-c2361de63250") : secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:08.832112 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:08.832028    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 10:30:08.832112 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:08.832092    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert podName:4f9527ed-c3ba-45eb-a51c-80922f29870a nodeName:}" failed. No retries permitted until 2026-09-09 10:30:10.83207546 +0000 UTC m=+252.887634511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jpm9t" (UID: "4f9527ed-c3ba-45eb-a51c-80922f29870a") : secret "networking-console-plugin-cert" not found
Sep 09 10:30:09.173056 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:09.173014    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w" event={"ID":"018e1efe-ab8e-4a08-a366-56aab8b91458","Type":"ContainerStarted","Data":"87f30349c89e82507fc459e4b277dcb04c596ceb500ea23bc485962077b3739d"}
Sep 09 10:30:09.189126 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:09.189069    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cwb4w" podStartSLOduration=1.9118871899999998 podStartE2EDuration="3.189047725s" totalImagesPullingTime="1.277160535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:09.187692316 +0000 UTC m=+251.243251385" watchObservedRunningTime="2026-09-09 10:30:09.189047725 +0000 UTC m=+251.244606776"
Sep 09 10:30:10.346672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:10.346625    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:30:10.347178 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:10.346818    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 10:30:10.347178 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:10.346926    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs podName:d68f3e9e-ee75-4e19-afde-af8bcf7c76d6 nodeName:}" failed. No retries permitted until 2026-09-09 10:32:12.346904197 +0000 UTC m=+374.402463240 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs") pod "network-metrics-daemon-d48rq" (UID: "d68f3e9e-ee75-4e19-afde-af8bcf7c76d6") : secret "metrics-daemon-secret" not found
Sep 09 10:30:10.849737 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:10.849680    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:10.849939 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:10.849858    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 10:30:10.849939 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:10.849910    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:10.850026 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:10.849960    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert podName:4f9527ed-c3ba-45eb-a51c-80922f29870a nodeName:}" failed. No retries permitted until 2026-09-09 10:30:14.849938424 +0000 UTC m=+256.905497458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jpm9t" (UID: "4f9527ed-c3ba-45eb-a51c-80922f29870a") : secret "networking-console-plugin-cert" not found
Sep 09 10:30:10.850026 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:10.849987    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:10.850026 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:10.850014    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 10:30:10.850147 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:10.850065    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls podName:a99b830c-9630-4af7-8f8c-1dbd9cb7c076 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:14.850049895 +0000 UTC m=+256.905608938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gqj4p" (UID: "a99b830c-9630-4af7-8f8c-1dbd9cb7c076") : secret "samples-operator-tls" not found
Sep 09 10:30:10.850147 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:10.850094    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:10.850147 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:10.850129    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls podName:fd142694-acce-48c2-ba76-c2361de63250 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:14.850118068 +0000 UTC m=+256.905677100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lm2ph" (UID: "fd142694-acce-48c2-ba76-c2361de63250") : secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:11.180050 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.180012    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz" event={"ID":"db6c7c40-96e8-473e-ae77-35168a674dbe","Type":"ContainerStarted","Data":"08eb3c09e327ceefefee95cbc8b67bb7c4060d8a984afcd12aa7c3be1b26f68e"}
Sep 09 10:30:11.181524 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.181499    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6" event={"ID":"572133a3-bac1-458a-b2ed-fb140a33bbe3","Type":"ContainerStarted","Data":"1de0e8264c0c4e276d6a12657e52d3ae9f12a56ab2153cbbfb7ac80d12c88db8"}
Sep 09 10:30:11.197131 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.197081    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz" podStartSLOduration=1.4804796439999999 podStartE2EDuration="4.197061397s" totalImagesPullingTime="2.716581753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:11.196474875 +0000 UTC m=+253.252033927" watchObservedRunningTime="2026-09-09 10:30:11.197061397 +0000 UTC m=+253.252620451"
Sep 09 10:30:11.476184 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.476088    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6" podStartSLOduration=2.594879664 podStartE2EDuration="5.476067882s" totalImagesPullingTime="2.881188218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:11.210648681 +0000 UTC m=+253.266207732" watchObservedRunningTime="2026-09-09 10:30:11.476067882 +0000 UTC m=+253.531626933"
Sep 09 10:30:11.476979 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.476960    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lpgj4"]
Sep 09 10:30:11.480294 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.480275    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.482652 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.482627    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 10:30:11.482723 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.482680    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89f9s\""
Sep 09 10:30:11.487029 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.483294    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 10:30:11.493311 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.493286    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lpgj4"]
Sep 09 10:30:11.657993 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.657959    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.658155 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.658003    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68170ad4-92bb-4332-aab6-831afb919a87-crio-socket\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.658155 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.658133    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj7px\" (UniqueName: \"kubernetes.io/projected/68170ad4-92bb-4332-aab6-831afb919a87-kube-api-access-nj7px\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.658230 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.658169    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68170ad4-92bb-4332-aab6-831afb919a87-data-volume\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.658230 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.658185    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.759086 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.759011    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68170ad4-92bb-4332-aab6-831afb919a87-data-volume\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.759086 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.759048    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.759248 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:11.759143    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:11.759248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.759211    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.759319 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.759245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68170ad4-92bb-4332-aab6-831afb919a87-crio-socket\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.759370 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:11.759320    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:12.259280274 +0000 UTC m=+254.314839318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:11.759370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.759325    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68170ad4-92bb-4332-aab6-831afb919a87-crio-socket\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.759370 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:11.759357    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:11.759482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.759381    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68170ad4-92bb-4332-aab6-831afb919a87-data-volume\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.759482 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:11.759409    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:12.259393429 +0000 UTC m=+254.314952462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:11.759482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.759452    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj7px\" (UniqueName: \"kubernetes.io/projected/68170ad4-92bb-4332-aab6-831afb919a87-kube-api-access-nj7px\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:11.771299 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:11.771274    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj7px\" (UniqueName: \"kubernetes.io/projected/68170ad4-92bb-4332-aab6-831afb919a87-kube-api-access-nj7px\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:12.263417 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.263381    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:12.263562 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.263470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:12.263562 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:12.263475    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:12.263562 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:12.263539    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:13.263520591 +0000 UTC m=+255.319079633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:12.263682 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:12.263589    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:12.263682 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:12.263634    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:13.26362347 +0000 UTC m=+255.319182499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:12.736589 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.736556    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6"]
Sep 09 10:30:12.739429 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.739413    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6"
Sep 09 10:30:12.741510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.741489    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 10:30:12.742279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.742255    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zcrvm\""
Sep 09 10:30:12.742279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.742276    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 10:30:12.747595 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.747574    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6"]
Sep 09 10:30:12.868768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.868731    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdtx\" (UniqueName: \"kubernetes.io/projected/0e769117-bd36-419b-aacf-70a10ce16afe-kube-api-access-jgdtx\") pod \"migrator-fdbb8fdff-4xtr6\" (UID: \"0e769117-bd36-419b-aacf-70a10ce16afe\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6"
Sep 09 10:30:12.970127 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.970091    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdtx\" (UniqueName: \"kubernetes.io/projected/0e769117-bd36-419b-aacf-70a10ce16afe-kube-api-access-jgdtx\") pod \"migrator-fdbb8fdff-4xtr6\" (UID: \"0e769117-bd36-419b-aacf-70a10ce16afe\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6"
Sep 09 10:30:12.978942 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:12.978910    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdtx\" (UniqueName: \"kubernetes.io/projected/0e769117-bd36-419b-aacf-70a10ce16afe-kube-api-access-jgdtx\") pod \"migrator-fdbb8fdff-4xtr6\" (UID: \"0e769117-bd36-419b-aacf-70a10ce16afe\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6"
Sep 09 10:30:13.048091 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:13.048025    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6"
Sep 09 10:30:13.272036 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:13.272002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:13.272213 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:13.272112    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:13.272213 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:13.272173    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:13.272334 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:13.272214    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:13.272334 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:13.272258    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:15.272236665 +0000 UTC m=+257.327795693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:13.272334 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:13.272280    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:15.272270297 +0000 UTC m=+257.327829331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:13.318192 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:13.318167    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6"]
Sep 09 10:30:13.320526 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:13.320482    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e769117_bd36_419b_aacf_70a10ce16afe.slice/crio-1dade7c39b4f04f9c07f98a70053383c3956c305aa9b87770f9360c506605d35 WatchSource:0}: Error finding container 1dade7c39b4f04f9c07f98a70053383c3956c305aa9b87770f9360c506605d35: Status 404 returned error can't find the container with id 1dade7c39b4f04f9c07f98a70053383c3956c305aa9b87770f9360c506605d35
Sep 09 10:30:13.638176 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:13.638150    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flzz6_4737cd63-ad67-44bf-9164-f1a50aa8eda7/dns-node-resolver/0.log"
Sep 09 10:30:14.169458 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.169423    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-789nn"]
Sep 09 10:30:14.173774 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.173757    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.175928 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.175904    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ds9b2\""
Sep 09 10:30:14.176057 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.176040    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 10:30:14.176131 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.176116    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 10:30:14.176198 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.176144    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 10:30:14.176198 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.176145    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 10:30:14.178193 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.178169    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6j74\" (UniqueName: \"kubernetes.io/projected/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-kube-api-access-d6j74\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.178846 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.178302    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-signing-cabundle\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.178846 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.178383    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-signing-key\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.179425 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.179405    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-789nn"]
Sep 09 10:30:14.192108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.192084    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6" event={"ID":"0e769117-bd36-419b-aacf-70a10ce16afe","Type":"ContainerStarted","Data":"1dade7c39b4f04f9c07f98a70053383c3956c305aa9b87770f9360c506605d35"}
Sep 09 10:30:14.279394 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.279367    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-signing-cabundle\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.279549 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.279431    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-signing-key\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.279549 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.279462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6j74\" (UniqueName: \"kubernetes.io/projected/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-kube-api-access-d6j74\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.279976 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.279954    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-signing-cabundle\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.281761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.281740    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-signing-key\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.288564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.288542    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6j74\" (UniqueName: \"kubernetes.io/projected/e88e43c7-fdf6-46d4-aad2-f0895cd202ef-kube-api-access-d6j74\") pod \"service-ca-75d99c484-789nn\" (UID: \"e88e43c7-fdf6-46d4-aad2-f0895cd202ef\") " pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.483036 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.482958    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-789nn"
Sep 09 10:30:14.639715 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.639688    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dl5gz_c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886/node-ca/0.log"
Sep 09 10:30:14.756520 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.756432    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-789nn"]
Sep 09 10:30:14.761084 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:14.761047    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88e43c7_fdf6_46d4_aad2_f0895cd202ef.slice/crio-0c466b322939c1c064a4da16ea3c19569431cf5b0a78830d72302e4a1d86dc62 WatchSource:0}: Error finding container 0c466b322939c1c064a4da16ea3c19569431cf5b0a78830d72302e4a1d86dc62: Status 404 returned error can't find the container with id 0c466b322939c1c064a4da16ea3c19569431cf5b0a78830d72302e4a1d86dc62
Sep 09 10:30:14.884629 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.884593    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:14.884808 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.884662    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:14.884808 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:14.884764    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:14.884808 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:14.884789    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 10:30:14.885022 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:14.884832    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 10:30:14.885022 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:14.884767    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:14.885022 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:14.884905    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert podName:4f9527ed-c3ba-45eb-a51c-80922f29870a nodeName:}" failed. No retries permitted until 2026-09-09 10:30:22.884829409 +0000 UTC m=+264.940388450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jpm9t" (UID: "4f9527ed-c3ba-45eb-a51c-80922f29870a") : secret "networking-console-plugin-cert" not found
Sep 09 10:30:14.885022 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:14.884951    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls podName:a99b830c-9630-4af7-8f8c-1dbd9cb7c076 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:22.88492459 +0000 UTC m=+264.940483632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gqj4p" (UID: "a99b830c-9630-4af7-8f8c-1dbd9cb7c076") : secret "samples-operator-tls" not found
Sep 09 10:30:14.885022 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:14.884982    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls podName:fd142694-acce-48c2-ba76-c2361de63250 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:22.884970528 +0000 UTC m=+264.940529560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lm2ph" (UID: "fd142694-acce-48c2-ba76-c2361de63250") : secret "cluster-monitoring-operator-tls" not found
Sep 09 10:30:15.197736 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:15.197705    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-789nn" event={"ID":"e88e43c7-fdf6-46d4-aad2-f0895cd202ef","Type":"ContainerStarted","Data":"30d46ee3cb4b38e9c1723bfaff2765bb49c41e11e3624c0e9236f54fee190569"}
Sep 09 10:30:15.198101 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:15.197767    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-789nn" event={"ID":"e88e43c7-fdf6-46d4-aad2-f0895cd202ef","Type":"ContainerStarted","Data":"0c466b322939c1c064a4da16ea3c19569431cf5b0a78830d72302e4a1d86dc62"}
Sep 09 10:30:15.225875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:15.225828    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-789nn" podStartSLOduration=1.225812486 podStartE2EDuration="1.225812486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:15.224858963 +0000 UTC m=+257.280418014" watchObservedRunningTime="2026-09-09 10:30:15.225812486 +0000 UTC m=+257.281371573"
Sep 09 10:30:15.289802 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:15.289770    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:15.289930 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:15.289910    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:15.290006 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:15.289982    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:15.290148 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:15.290127    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:15.290218 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:15.290194    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:19.290158173 +0000 UTC m=+261.345717207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:15.290278 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:15.290229    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:19.290216442 +0000 UTC m=+261.345775477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:15.439585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:15.439546    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zznrb_ad647ae4-c0ad-4e78-a358-cbfb757329f0/serve-healthcheck-canary/0.log"
Sep 09 10:30:16.202576 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:16.202543    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6" event={"ID":"0e769117-bd36-419b-aacf-70a10ce16afe","Type":"ContainerStarted","Data":"062f1e7f39a0066485499aa40a138f65cc8fb259196a626ab494d51f855ddd1b"}
Sep 09 10:30:16.203018 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:16.202598    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6" event={"ID":"0e769117-bd36-419b-aacf-70a10ce16afe","Type":"ContainerStarted","Data":"9fd35ef9ed2cfc7c09efd6157d762e723f0f4406df0ab83cb40c9fa51e7f59c1"}
Sep 09 10:30:16.226837 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:16.226787    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4xtr6" podStartSLOduration=2.350773525 podStartE2EDuration="4.22676956s" totalImagesPullingTime="1.875996035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:16.226382042 +0000 UTC m=+258.281941102" watchObservedRunningTime="2026-09-09 10:30:16.22676956 +0000 UTC m=+258.282328611"
Sep 09 10:30:19.325325 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:19.325291    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:19.325707 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:19.325453    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:19.325707 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:19.325509    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 10:30:19.325707 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:19.325459    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:19.325707 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:19.325518    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:27.325502748 +0000 UTC m=+269.381061803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : secret "insights-runtime-extractor-tls" not found
Sep 09 10:30:19.325707 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:19.325588    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm podName:68170ad4-92bb-4332-aab6-831afb919a87 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:27.325566613 +0000 UTC m=+269.381125644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lpgj4" (UID: "68170ad4-92bb-4332-aab6-831afb919a87") : configmap "kube-rbac-proxy" not found
Sep 09 10:30:22.959331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:22.959233    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:22.959331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:22.959326    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:22.959838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:22.959349    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:22.959838 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:22.959398    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 10:30:22.959838 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:22.959478    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert podName:4f9527ed-c3ba-45eb-a51c-80922f29870a nodeName:}" failed. No retries permitted until 2026-09-09 10:30:38.959461872 +0000 UTC m=+281.015020900 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jpm9t" (UID: "4f9527ed-c3ba-45eb-a51c-80922f29870a") : secret "networking-console-plugin-cert" not found
Sep 09 10:30:22.961808 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:22.961783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a99b830c-9630-4af7-8f8c-1dbd9cb7c076-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gqj4p\" (UID: \"a99b830c-9630-4af7-8f8c-1dbd9cb7c076\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:22.961948 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:22.961927    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd142694-acce-48c2-ba76-c2361de63250-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lm2ph\" (UID: \"fd142694-acce-48c2-ba76-c2361de63250\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:22.978751 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:22.978727    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"
Sep 09 10:30:22.991626 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:22.991605    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"
Sep 09 10:30:23.253469 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:23.253389    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p"]
Sep 09 10:30:23.279300 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:23.279276    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph"]
Sep 09 10:30:23.281658 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:23.281629    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd142694_acce_48c2_ba76_c2361de63250.slice/crio-2907b5c88a05ca0bff2eb2d14ab716b8fbfb73cd729edf8b6af384342df2a945 WatchSource:0}: Error finding container 2907b5c88a05ca0bff2eb2d14ab716b8fbfb73cd729edf8b6af384342df2a945: Status 404 returned error can't find the container with id 2907b5c88a05ca0bff2eb2d14ab716b8fbfb73cd729edf8b6af384342df2a945
Sep 09 10:30:24.230699 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:24.230648    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p" event={"ID":"a99b830c-9630-4af7-8f8c-1dbd9cb7c076","Type":"ContainerStarted","Data":"4bae85aa09a9f9a34eb609f18f5e827f4fd74d3a15f0e77c54583caefb78e9fd"}
Sep 09 10:30:24.232197 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:24.232166    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph" event={"ID":"fd142694-acce-48c2-ba76-c2361de63250","Type":"ContainerStarted","Data":"2907b5c88a05ca0bff2eb2d14ab716b8fbfb73cd729edf8b6af384342df2a945"}
Sep 09 10:30:25.237826 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:25.237788    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph" event={"ID":"fd142694-acce-48c2-ba76-c2361de63250","Type":"ContainerStarted","Data":"61fea442efbc884144be7a1d66d3e54c4176988632fcd200219ab764fa9cd72f"}
Sep 09 10:30:25.257776 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:25.257723    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lm2ph" podStartSLOduration=16.437158889 podStartE2EDuration="18.257702624s" totalImagesPullingTime="1.820543735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:25.254936281 +0000 UTC m=+267.310495332" watchObservedRunningTime="2026-09-09 10:30:25.257702624 +0000 UTC m=+267.313261673"
Sep 09 10:30:26.242065 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:26.242030    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p" event={"ID":"a99b830c-9630-4af7-8f8c-1dbd9cb7c076","Type":"ContainerStarted","Data":"d33801ee699f38df8df579cc2d11dc4d027afcbd22845750e18b962155e28bcc"}
Sep 09 10:30:26.242453 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:26.242105    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p" event={"ID":"a99b830c-9630-4af7-8f8c-1dbd9cb7c076","Type":"ContainerStarted","Data":"8d44850deb7859a4d274c09ff36ceebf448fc3526abf3450f51377ae91a1f3ea"}
Sep 09 10:30:26.260618 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:26.260572    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gqj4p" podStartSLOduration=16.889737742 podStartE2EDuration="19.260557769s" totalImagesPullingTime="2.370820027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:26.259006125 +0000 UTC m=+268.314565185" watchObservedRunningTime="2026-09-09 10:30:26.260557769 +0000 UTC m=+268.316116818"
Sep 09 10:30:27.397244 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:27.397206    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:27.397609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:27.397285    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:27.397742 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:27.397723    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68170ad4-92bb-4332-aab6-831afb919a87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:27.399492 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:27.399476    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68170ad4-92bb-4332-aab6-831afb919a87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lpgj4\" (UID: \"68170ad4-92bb-4332-aab6-831afb919a87\") " pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:27.694252 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:27.694157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lpgj4"
Sep 09 10:30:27.962298 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:27.962269    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lpgj4"]
Sep 09 10:30:27.964568 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:27.964539    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68170ad4_92bb_4332_aab6_831afb919a87.slice/crio-277fe6355fa099256352c61a0b4e6677d309e8a435fd02ef6457b07ecc2ef550 WatchSource:0}: Error finding container 277fe6355fa099256352c61a0b4e6677d309e8a435fd02ef6457b07ecc2ef550: Status 404 returned error can't find the container with id 277fe6355fa099256352c61a0b4e6677d309e8a435fd02ef6457b07ecc2ef550
Sep 09 10:30:28.249007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:28.248925    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lpgj4" event={"ID":"68170ad4-92bb-4332-aab6-831afb919a87","Type":"ContainerStarted","Data":"5b008d0ec8e1a788958ee07539aae330493c9df0a871f73aaf1499adf692f9b1"}
Sep 09 10:30:28.249007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:28.248981    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lpgj4" event={"ID":"68170ad4-92bb-4332-aab6-831afb919a87","Type":"ContainerStarted","Data":"277fe6355fa099256352c61a0b4e6677d309e8a435fd02ef6457b07ecc2ef550"}
Sep 09 10:30:29.254094 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:29.254055    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lpgj4" event={"ID":"68170ad4-92bb-4332-aab6-831afb919a87","Type":"ContainerStarted","Data":"a1e2a70dc76eb5f20f6ace7cac5d635b9d79593a7277974a882e6937f65fd4ef"}
Sep 09 10:30:31.262864 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:31.262829    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lpgj4" event={"ID":"68170ad4-92bb-4332-aab6-831afb919a87","Type":"ContainerStarted","Data":"6f114c4e7c55a406bf1aa4a8403e2ba83d85fdb3afa6bcc2a1485e4640e21ed1"}
Sep 09 10:30:31.281638 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:31.281583    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lpgj4" podStartSLOduration=17.900235778 podStartE2EDuration="20.281565477s" totalImagesPullingTime="2.381329699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:30:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:31.280758366 +0000 UTC m=+273.336317416" watchObservedRunningTime="2026-09-09 10:30:31.281565477 +0000 UTC m=+273.337124527"
Sep 09 10:30:37.938351 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:37.938308    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4c2kf" podUID="4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226"
Sep 09 10:30:37.938351 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:37.938308    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" podUID="775fc55b-45cc-4506-b06e-822381628930"
Sep 09 10:30:38.286649 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.286580    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:30:38.286780 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.286581    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:30:38.414887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.414859    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7597dccf8d-4lcwk"]
Sep 09 10:30:38.419127 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.419112    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.421157 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.421137    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 10:30:38.421403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.421387    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 10:30:38.421782 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.421768    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 10:30:38.422035 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.422014    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 10:30:38.422148 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.422085    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 10:30:38.422148 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.422110    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 10:30:38.422253 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.422149    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dchvh\""
Sep 09 10:30:38.422253 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.422085    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 10:30:38.428662 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.428639    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7597dccf8d-4lcwk"]
Sep 09 10:30:38.480510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.480485    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-oauth-serving-cert\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.480640 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.480517    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-service-ca\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.480640 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.480588    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-config\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.480721 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.480658    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-serving-cert\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.480721 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.480687    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-oauth-config\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.480802 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.480729    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r7xm\" (UniqueName: \"kubernetes.io/projected/b210a91c-3977-4e96-8688-6f3a4d6a291c-kube-api-access-5r7xm\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.582173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-oauth-serving-cert\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.582173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-service-ca\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.582362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582258    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-config\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.582362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582315    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-serving-cert\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.582458 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-oauth-config\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.582500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582484    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7xm\" (UniqueName: \"kubernetes.io/projected/b210a91c-3977-4e96-8688-6f3a4d6a291c-kube-api-access-5r7xm\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.582924 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582879    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-service-ca\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.583031 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582937    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-oauth-serving-cert\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.583031 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.582946    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-config\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.584737 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.584706    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-oauth-config\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.584737 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.584730    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-serving-cert\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.590376 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.590350    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7xm\" (UniqueName: \"kubernetes.io/projected/b210a91c-3977-4e96-8688-6f3a4d6a291c-kube-api-access-5r7xm\") pod \"console-7597dccf8d-4lcwk\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") " pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.728025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.727989    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:38.986342 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.986299    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:38.988714 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.988686    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f9527ed-c3ba-45eb-a51c-80922f29870a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jpm9t\" (UID: \"4f9527ed-c3ba-45eb-a51c-80922f29870a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:38.998420 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:38.998397    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7597dccf8d-4lcwk"]
Sep 09 10:30:39.000612 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:39.000589    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb210a91c_3977_4e96_8688_6f3a4d6a291c.slice/crio-a2fd3c435c366e03848d0a28e37a86b6389517047148e48cd829f9ec54d3273f WatchSource:0}: Error finding container a2fd3c435c366e03848d0a28e37a86b6389517047148e48cd829f9ec54d3273f: Status 404 returned error can't find the container with id a2fd3c435c366e03848d0a28e37a86b6389517047148e48cd829f9ec54d3273f
Sep 09 10:30:39.170733 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:39.170707    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t"
Sep 09 10:30:39.291275 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:39.291195    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7597dccf8d-4lcwk" event={"ID":"b210a91c-3977-4e96-8688-6f3a4d6a291c","Type":"ContainerStarted","Data":"a2fd3c435c366e03848d0a28e37a86b6389517047148e48cd829f9ec54d3273f"}
Sep 09 10:30:39.437517 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:39.437454    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jpm9t"]
Sep 09 10:30:39.439922 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:39.439877    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9527ed_c3ba_45eb_a51c_80922f29870a.slice/crio-8f5e117becb96c8501ef1f23a9df0a4410fde256faa673e2bfa327fbd0fbd96d WatchSource:0}: Error finding container 8f5e117becb96c8501ef1f23a9df0a4410fde256faa673e2bfa327fbd0fbd96d: Status 404 returned error can't find the container with id 8f5e117becb96c8501ef1f23a9df0a4410fde256faa673e2bfa327fbd0fbd96d
Sep 09 10:30:40.296412 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:40.296378    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t" event={"ID":"4f9527ed-c3ba-45eb-a51c-80922f29870a","Type":"ContainerStarted","Data":"8f5e117becb96c8501ef1f23a9df0a4410fde256faa673e2bfa327fbd0fbd96d"}
Sep 09 10:30:41.303644 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:41.303607    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:30:41.304101 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:41.303656    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:30:41.306452 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:41.306423    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"image-registry-5b656c84bb-9hx2l\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") " pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:30:41.306596 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:41.306423    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226-metrics-tls\") pod \"dns-default-4c2kf\" (UID: \"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226\") " pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:30:41.590088 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:41.589994    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-272r8\""
Sep 09 10:30:41.590088 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:41.589994    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4xzkl\""
Sep 09 10:30:41.597514 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:41.597483    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:30:41.597694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:41.597651    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:30:42.005076 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.005012    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4c2kf"]
Sep 09 10:30:42.006140 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.006117    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b656c84bb-9hx2l"]
Sep 09 10:30:42.007487 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:42.007458    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2d6d3d_fb6c_4aa1_b70e_839cf96a3226.slice/crio-959d82f8ec93fb9e17817e0d68ca31ae8a306609829c8324f83cb919514420fc WatchSource:0}: Error finding container 959d82f8ec93fb9e17817e0d68ca31ae8a306609829c8324f83cb919514420fc: Status 404 returned error can't find the container with id 959d82f8ec93fb9e17817e0d68ca31ae8a306609829c8324f83cb919514420fc
Sep 09 10:30:42.008852 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:42.008804    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775fc55b_45cc_4506_b06e_822381628930.slice/crio-03e800e5866127b23c23b130409d7386ab0b8fc3c1e51e491d8304eccc339315 WatchSource:0}: Error finding container 03e800e5866127b23c23b130409d7386ab0b8fc3c1e51e491d8304eccc339315: Status 404 returned error can't find the container with id 03e800e5866127b23c23b130409d7386ab0b8fc3c1e51e491d8304eccc339315
Sep 09 10:30:42.305791 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.305752    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7597dccf8d-4lcwk" event={"ID":"b210a91c-3977-4e96-8688-6f3a4d6a291c","Type":"ContainerStarted","Data":"aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09"}
Sep 09 10:30:42.307402 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.307361    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4c2kf" event={"ID":"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226","Type":"ContainerStarted","Data":"959d82f8ec93fb9e17817e0d68ca31ae8a306609829c8324f83cb919514420fc"}
Sep 09 10:30:42.309678 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.309195    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" event={"ID":"775fc55b-45cc-4506-b06e-822381628930","Type":"ContainerStarted","Data":"21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6"}
Sep 09 10:30:42.309678 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.309241    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" event={"ID":"775fc55b-45cc-4506-b06e-822381628930","Type":"ContainerStarted","Data":"03e800e5866127b23c23b130409d7386ab0b8fc3c1e51e491d8304eccc339315"}
Sep 09 10:30:42.309678 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.309352    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:30:42.324867 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.324812    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7597dccf8d-4lcwk" podStartSLOduration=1.535952582 podStartE2EDuration="4.324797397s" totalImagesPullingTime="2.788844815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:42.323045273 +0000 UTC m=+284.378604346" watchObservedRunningTime="2026-09-09 10:30:42.324797397 +0000 UTC m=+284.380356447"
Sep 09 10:30:42.342259 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:42.342208    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" podStartSLOduration=283.342193758 podStartE2EDuration="4m43.342193758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:42.340786685 +0000 UTC m=+284.396345748" watchObservedRunningTime="2026-09-09 10:30:42.342193758 +0000 UTC m=+284.397752808"
Sep 09 10:30:43.316230 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:43.316185    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t" event={"ID":"4f9527ed-c3ba-45eb-a51c-80922f29870a","Type":"ContainerStarted","Data":"96a3f61ea6187ee539efce84c7e96f08cd423155b8a32189d1b911c3f25b1fbf"}
Sep 09 10:30:43.332599 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:43.332537    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-jpm9t" podStartSLOduration=33.30198915 podStartE2EDuration="36.332522979s" totalImagesPullingTime="3.030533829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:43.332490121 +0000 UTC m=+285.388049171" watchObservedRunningTime="2026-09-09 10:30:43.332522979 +0000 UTC m=+285.388082041"
Sep 09 10:30:45.324428 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:45.324395    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4c2kf" event={"ID":"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226","Type":"ContainerStarted","Data":"8dbe338e9dba694c82bb848a24bd37697de5b107dab51049c5111d68911c16fb"}
Sep 09 10:30:45.324816 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:45.324448    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4c2kf" event={"ID":"4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226","Type":"ContainerStarted","Data":"d90ad27d4ff184b201388085829804abb17bcd8f353e4d58322069897f048ff0"}
Sep 09 10:30:45.324816 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:45.324483    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:30:45.342798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:45.342745    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4c2kf" podStartSLOduration=251.962294648 podStartE2EDuration="4m14.342731083s" totalImagesPullingTime="2.380436435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:45.341976256 +0000 UTC m=+287.397535298" watchObservedRunningTime="2026-09-09 10:30:45.342731083 +0000 UTC m=+287.398290130"
Sep 09 10:30:47.085843 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.085804    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gjrpb"]
Sep 09 10:30:47.089234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.089205    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.092380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.092332    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 10:30:47.092380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.092348    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x8vlp\""
Sep 09 10:30:47.092575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.092438    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 10:30:47.092575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.092332    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 10:30:47.092575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.092346    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 10:30:47.256877 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.256839    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21911b8f-8acf-446d-a04f-b5f367928408-metrics-client-ca\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.256877 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.256879    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.257119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.256908    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-textfile\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.257119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.257044    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.257119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.257080    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5nt\" (UniqueName: \"kubernetes.io/projected/21911b8f-8acf-446d-a04f-b5f367928408-kube-api-access-rc5nt\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.257248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.257129    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-wtmp\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.257248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.257157    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-tls\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.257248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.257181    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-root\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.257379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.257247    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-sys\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.357738 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.357657    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21911b8f-8acf-446d-a04f-b5f367928408-metrics-client-ca\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.357738 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.357696    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.357738 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.357722    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-textfile\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.357803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.357832    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5nt\" (UniqueName: \"kubernetes.io/projected/21911b8f-8acf-446d-a04f-b5f367928408-kube-api-access-rc5nt\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.357863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-wtmp\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.357883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-tls\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.357929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-root\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358236 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.358007    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-sys\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358236 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.358118    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-sys\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358236 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.358147    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-textfile\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358236 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.358205    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-root\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358366 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.358246    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-wtmp\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358429 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.358357    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21911b8f-8acf-446d-a04f-b5f367928408-metrics-client-ca\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.358593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.358574    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.360218 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.360188    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.360414 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.360396    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21911b8f-8acf-446d-a04f-b5f367928408-node-exporter-tls\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.367143 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.367121    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5nt\" (UniqueName: \"kubernetes.io/projected/21911b8f-8acf-446d-a04f-b5f367928408-kube-api-access-rc5nt\") pod \"node-exporter-gjrpb\" (UID: \"21911b8f-8acf-446d-a04f-b5f367928408\") " pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.399266 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:47.399243    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gjrpb"
Sep 09 10:30:47.411171 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:47.411143    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21911b8f_8acf_446d_a04f_b5f367928408.slice/crio-6c9cb93d501c0e5ea9cf02d989377a045e028ae3ba2ed8b6ac37554fe55a0d94 WatchSource:0}: Error finding container 6c9cb93d501c0e5ea9cf02d989377a045e028ae3ba2ed8b6ac37554fe55a0d94: Status 404 returned error can't find the container with id 6c9cb93d501c0e5ea9cf02d989377a045e028ae3ba2ed8b6ac37554fe55a0d94
Sep 09 10:30:48.157867 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.157830    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:30:48.161976 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.161952    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.174122 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.174098    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 10:30:48.174844 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.174594    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 10:30:48.174844 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.174704    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nhjq2\""
Sep 09 10:30:48.174844 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.174828    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 10:30:48.175557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.175541    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 10:30:48.175622 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.175594    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 10:30:48.175878 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.175858    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 10:30:48.175993 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.175868    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 10:30:48.176934 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.176918    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 10:30:48.179116 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.179100    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 10:30:48.227976 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.227944    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:30:48.266404 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266375    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.266585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266410    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.266585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266432    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-volume\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.266585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266454    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.266585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266482    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-out\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.266585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266511    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.266585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266534    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-web-config\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.266585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266562    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.267002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266648    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.267002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266713    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.267002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266741    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.267002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266758    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.267002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.266779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qm9m\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-kube-api-access-6qm9m\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.337028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.336988    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjrpb" event={"ID":"21911b8f-8acf-446d-a04f-b5f367928408","Type":"ContainerStarted","Data":"6c9cb93d501c0e5ea9cf02d989377a045e028ae3ba2ed8b6ac37554fe55a0d94"}
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367515    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367561    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367590    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367619    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qm9m\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-kube-api-access-6qm9m\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367673    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367710    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367735    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-volume\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367761    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367788    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-out\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367812    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367840    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-web-config\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367877    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.367951    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.368067    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:48.368175    2586 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 10:30:48.368314 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:48.368243    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls podName:60f808ac-2802-4c98-92b9-8a41c7ebe481 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:48.868225275 +0000 UTC m=+290.923784317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481") : secret "alertmanager-main-tls" not found
Sep 09 10:30:48.369626 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.369314    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.372534 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.372495    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.372770 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.372745    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.373173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.373121    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-volume\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.373367 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.373337    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-out\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.373857 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.373834    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.373973 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.373858    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.374326 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.374302    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-web-config\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.376351 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.376325    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.379422 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.379396    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qm9m\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-kube-api-access-6qm9m\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.379939 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.379835    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.728340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.728304    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:48.728504 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.728393    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:30:48.729598 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.729577    2586 patch_prober.go:28] interesting pod/console-7597dccf8d-4lcwk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body=
Sep 09 10:30:48.729657 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.729619    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7597dccf8d-4lcwk" podUID="b210a91c-3977-4e96-8688-6f3a4d6a291c" containerName="console" probeResult="failure" output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused"
Sep 09 10:30:48.872436 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.872349    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:48.874806 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:48.874779    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:49.074232 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:49.074193    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:30:49.342523 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:49.342438    2586 generic.go:356] "Generic (PLEG): container finished" podID="21911b8f-8acf-446d-a04f-b5f367928408" containerID="09376642e09fb373ce6e58eb7eb9f7a5fd56bf3185b247da7c8e87ba9acea201" exitCode=0
Sep 09 10:30:49.342943 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:49.342530    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjrpb" event={"ID":"21911b8f-8acf-446d-a04f-b5f367928408","Type":"ContainerDied","Data":"09376642e09fb373ce6e58eb7eb9f7a5fd56bf3185b247da7c8e87ba9acea201"}
Sep 09 10:30:49.355160 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:49.355115    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:30:49.357123 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:49.357096    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f808ac_2802_4c98_92b9_8a41c7ebe481.slice/crio-1a39dade563c2aca2746dfc1eec6473dd5690b078c7f24a2e5c64f8b0f5a3056 WatchSource:0}: Error finding container 1a39dade563c2aca2746dfc1eec6473dd5690b078c7f24a2e5c64f8b0f5a3056: Status 404 returned error can't find the container with id 1a39dade563c2aca2746dfc1eec6473dd5690b078c7f24a2e5c64f8b0f5a3056
Sep 09 10:30:49.469270 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:30:49.469243    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-d48rq" podUID="d68f3e9e-ee75-4e19-afde-af8bcf7c76d6"
Sep 09 10:30:50.062212 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.062171    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"]
Sep 09 10:30:50.068011 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.067959    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.070577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.070549    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2ml25ha5rqf9c\""
Sep 09 10:30:50.070693 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.070636    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 10:30:50.070750 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.070560    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xkk7k\""
Sep 09 10:30:50.070786 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.070777    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 10:30:50.070920 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.070881    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 10:30:50.070966 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.070935    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 10:30:50.071093 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.071078    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 10:30:50.077080 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.076835    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"]
Sep 09 10:30:50.183843 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.183811    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-tls\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.184055 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.183874    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.184055 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.183994    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqxbk\" (UniqueName: \"kubernetes.io/projected/c688af9f-4254-46ce-89c1-732ca4651855-kube-api-access-vqxbk\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.184161 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.184078    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.184161 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.184121    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.184235 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.184163    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.184235 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.184192    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c688af9f-4254-46ce-89c1-732ca4651855-metrics-client-ca\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.184235 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.184223    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-grpc-tls\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.285251 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.285212    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.285251 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.285250    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c688af9f-4254-46ce-89c1-732ca4651855-metrics-client-ca\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.285491 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.285276    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-grpc-tls\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.285491 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.285439    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-tls\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.285594 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.285511    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.285594 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.285560    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqxbk\" (UniqueName: \"kubernetes.io/projected/c688af9f-4254-46ce-89c1-732ca4651855-kube-api-access-vqxbk\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.285695 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.285621    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.285695 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.285652    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.286171 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.286112    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c688af9f-4254-46ce-89c1-732ca4651855-metrics-client-ca\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.288556 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.288507    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.288700 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.288672    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.288812 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.288718    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-grpc-tls\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.288901 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.288865    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.288958 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.288867    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-tls\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.289306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.289284    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c688af9f-4254-46ce-89c1-732ca4651855-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.293394 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.293373    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqxbk\" (UniqueName: \"kubernetes.io/projected/c688af9f-4254-46ce-89c1-732ca4651855-kube-api-access-vqxbk\") pod \"thanos-querier-6b975d59c4-qp5jn\" (UID: \"c688af9f-4254-46ce-89c1-732ca4651855\") " pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.348444 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.348346    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjrpb" event={"ID":"21911b8f-8acf-446d-a04f-b5f367928408","Type":"ContainerStarted","Data":"c102610136d14fae0c021173b72ffd286596806c7d6d820b168ccc2f07cdd6a9"}
Sep 09 10:30:50.348444 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.348427    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjrpb" event={"ID":"21911b8f-8acf-446d-a04f-b5f367928408","Type":"ContainerStarted","Data":"caea8b3a3987531f4178fe40d3847504efc2f6b60b5340b130d349d293f6e7c9"}
Sep 09 10:30:50.349662 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.349638    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerStarted","Data":"1a39dade563c2aca2746dfc1eec6473dd5690b078c7f24a2e5c64f8b0f5a3056"}
Sep 09 10:30:50.367557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.367510    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gjrpb" podStartSLOduration=2.2182136359999998 podStartE2EDuration="3.367492204s" totalImagesPullingTime="1.086705692s" totalInitContainerRuntime="62.572876ms" isStatefulPod=true podCreationTimestamp="2026-09-09 10:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:50.366859823 +0000 UTC m=+292.422418894" watchObservedRunningTime="2026-09-09 10:30:50.367492204 +0000 UTC m=+292.423051258"
Sep 09 10:30:50.380947 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.380924    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:50.667621 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:50.667592    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"]
Sep 09 10:30:50.670610 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:50.670583    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc688af9f_4254_46ce_89c1_732ca4651855.slice/crio-9326b13e7a3bed77ae673928c82591d98067a610a626c6e88c0cc850a76fd7d3 WatchSource:0}: Error finding container 9326b13e7a3bed77ae673928c82591d98067a610a626c6e88c0cc850a76fd7d3: Status 404 returned error can't find the container with id 9326b13e7a3bed77ae673928c82591d98067a610a626c6e88c0cc850a76fd7d3
Sep 09 10:30:51.355615 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.355580    2586 generic.go:356] "Generic (PLEG): container finished" podID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerID="8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e" exitCode=0
Sep 09 10:30:51.356082 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.355676    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerDied","Data":"8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e"}
Sep 09 10:30:51.357290 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.357251    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn" event={"ID":"c688af9f-4254-46ce-89c1-732ca4651855","Type":"ContainerStarted","Data":"9326b13e7a3bed77ae673928c82591d98067a610a626c6e88c0cc850a76fd7d3"}
Sep 09 10:30:51.398343 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.398314    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-588d844b79-6xj8d"]
Sep 09 10:30:51.401743 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.401718    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.405294 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.405272    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6tvs8puqisake\""
Sep 09 10:30:51.406367 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.406348    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 10:30:51.406478 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.406413    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 10:30:51.406831 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.406806    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 10:30:51.409806 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.407499    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 10:30:51.411737 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.411715    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gjbnk\""
Sep 09 10:30:51.415710 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.415685    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-588d844b79-6xj8d"]
Sep 09 10:30:51.498172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.498134    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-client-ca-bundle\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.498343 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.498194    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9848bce8-6ce5-47a3-839d-ebd153b2ed07-metrics-server-audit-profiles\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.498343 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.498280    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-secret-metrics-server-tls\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.498458 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.498364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9848bce8-6ce5-47a3-839d-ebd153b2ed07-audit-log\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.499172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.498913    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9848bce8-6ce5-47a3-839d-ebd153b2ed07-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.499172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.499071    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-secret-metrics-server-client-certs\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.499172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.499103    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkfbg\" (UniqueName: \"kubernetes.io/projected/9848bce8-6ce5-47a3-839d-ebd153b2ed07-kube-api-access-qkfbg\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.599834 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.599792    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9848bce8-6ce5-47a3-839d-ebd153b2ed07-metrics-server-audit-profiles\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.600015 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.599933    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-secret-metrics-server-tls\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.600015 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.599975    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9848bce8-6ce5-47a3-839d-ebd153b2ed07-audit-log\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.600134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.600024    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9848bce8-6ce5-47a3-839d-ebd153b2ed07-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.600134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.600094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-secret-metrics-server-client-certs\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.601034 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.600377    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9848bce8-6ce5-47a3-839d-ebd153b2ed07-audit-log\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.601034 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.600441    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkfbg\" (UniqueName: \"kubernetes.io/projected/9848bce8-6ce5-47a3-839d-ebd153b2ed07-kube-api-access-qkfbg\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.601034 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.600488    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-client-ca-bundle\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.601034 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.600914    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9848bce8-6ce5-47a3-839d-ebd153b2ed07-metrics-server-audit-profiles\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.601034 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.600944    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9848bce8-6ce5-47a3-839d-ebd153b2ed07-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.603208 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.603182    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-secret-metrics-server-client-certs\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.603359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.603333    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-secret-metrics-server-tls\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.603472 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.603370    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9848bce8-6ce5-47a3-839d-ebd153b2ed07-client-ca-bundle\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.609782 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.609734    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkfbg\" (UniqueName: \"kubernetes.io/projected/9848bce8-6ce5-47a3-839d-ebd153b2ed07-kube-api-access-qkfbg\") pod \"metrics-server-588d844b79-6xj8d\" (UID: \"9848bce8-6ce5-47a3-839d-ebd153b2ed07\") " pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.716000 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.715968    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:30:51.838652 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.838626    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"]
Sep 09 10:30:51.842263 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.842243    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:51.844245 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.844223    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 10:30:51.844371 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.844329    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cm7vb\""
Sep 09 10:30:51.850356 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.850333    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"]
Sep 09 10:30:51.853839 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.853818    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7597dccf8d-4lcwk"]
Sep 09 10:30:51.902864 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.902838    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66cfb7697-sjhff"]
Sep 09 10:30:51.906127 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.906109    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:51.914304 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.914280    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 10:30:51.921265 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:51.921242    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66cfb7697-sjhff"]
Sep 09 10:30:52.000860 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.000824    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-588d844b79-6xj8d"]
Sep 09 10:30:52.005180 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005157    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp5dl\" (UniqueName: \"kubernetes.io/projected/525c4513-e445-4135-9766-9bd1ef103e8a-kube-api-access-lp5dl\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.005301 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005220    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-serving-cert\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.005353 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005331    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-service-ca\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.005400 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005367    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-oauth-serving-cert\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.005400 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005389    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmhb\" (UniqueName: \"kubernetes.io/projected/d440b101-26a7-49e8-a737-c1b65eb56093-kube-api-access-8gmhb\") pod \"monitoring-plugin-85bbbbbf4-vhwp8\" (UID: \"d440b101-26a7-49e8-a737-c1b65eb56093\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:52.005502 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005480    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-console-config\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.005554 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005529    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-trusted-ca-bundle\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.005641 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005619    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d440b101-26a7-49e8-a737-c1b65eb56093-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-vhwp8\" (UID: \"d440b101-26a7-49e8-a737-c1b65eb56093\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:52.005705 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.005684    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-oauth-config\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.106539 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106500    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-service-ca\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.106689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106549    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-oauth-serving-cert\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.106689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106638    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmhb\" (UniqueName: \"kubernetes.io/projected/d440b101-26a7-49e8-a737-c1b65eb56093-kube-api-access-8gmhb\") pod \"monitoring-plugin-85bbbbbf4-vhwp8\" (UID: \"d440b101-26a7-49e8-a737-c1b65eb56093\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:52.106814 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106701    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-console-config\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.106814 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106730    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-trusted-ca-bundle\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.106814 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106780    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d440b101-26a7-49e8-a737-c1b65eb56093-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-vhwp8\" (UID: \"d440b101-26a7-49e8-a737-c1b65eb56093\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:52.106954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106823    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-oauth-config\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.106954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106870    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp5dl\" (UniqueName: \"kubernetes.io/projected/525c4513-e445-4135-9766-9bd1ef103e8a-kube-api-access-lp5dl\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.106954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.106946    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-serving-cert\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.107382 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.107329    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-service-ca\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.107543 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.107521    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-console-config\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.107622 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.107601    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-oauth-serving-cert\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.107915 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.107877    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-trusted-ca-bundle\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.110027 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.110005    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-oauth-config\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.110134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.110117    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-serving-cert\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.110196 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.110123    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d440b101-26a7-49e8-a737-c1b65eb56093-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-vhwp8\" (UID: \"d440b101-26a7-49e8-a737-c1b65eb56093\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:52.115845 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.115822    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmhb\" (UniqueName: \"kubernetes.io/projected/d440b101-26a7-49e8-a737-c1b65eb56093-kube-api-access-8gmhb\") pod \"monitoring-plugin-85bbbbbf4-vhwp8\" (UID: \"d440b101-26a7-49e8-a737-c1b65eb56093\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:52.115968 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.115947    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp5dl\" (UniqueName: \"kubernetes.io/projected/525c4513-e445-4135-9766-9bd1ef103e8a-kube-api-access-lp5dl\") pod \"console-66cfb7697-sjhff\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") " pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.152585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.152558    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:52.216920 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.216873    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:30:52.465075 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:52.464999    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9848bce8_6ce5_47a3_839d_ebd153b2ed07.slice/crio-00140a2440cc83f38bb2df23e53c209010cfe8a466aab0f274c3ecb6d4ec20fb WatchSource:0}: Error finding container 00140a2440cc83f38bb2df23e53c209010cfe8a466aab0f274c3ecb6d4ec20fb: Status 404 returned error can't find the container with id 00140a2440cc83f38bb2df23e53c209010cfe8a466aab0f274c3ecb6d4ec20fb
Sep 09 10:30:52.737969 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.737940    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"]
Sep 09 10:30:52.741216 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:52.741016    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66cfb7697-sjhff"]
Sep 09 10:30:52.741817 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:52.741787    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd440b101_26a7_49e8_a737_c1b65eb56093.slice/crio-2eeeb25a47b8d6526e1b320cae3322a9caff40f9f9d0505eb9a35c6c7feddc5e WatchSource:0}: Error finding container 2eeeb25a47b8d6526e1b320cae3322a9caff40f9f9d0505eb9a35c6c7feddc5e: Status 404 returned error can't find the container with id 2eeeb25a47b8d6526e1b320cae3322a9caff40f9f9d0505eb9a35c6c7feddc5e
Sep 09 10:30:52.745029 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:30:52.745001    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525c4513_e445_4135_9766_9bd1ef103e8a.slice/crio-6a99e2d0ad93a9a107c842b54f17f9e339f274eb0c25d6f0be99f463d371c1c4 WatchSource:0}: Error finding container 6a99e2d0ad93a9a107c842b54f17f9e339f274eb0c25d6f0be99f463d371c1c4: Status 404 returned error can't find the container with id 6a99e2d0ad93a9a107c842b54f17f9e339f274eb0c25d6f0be99f463d371c1c4
Sep 09 10:30:53.368059 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:53.368015    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8" event={"ID":"d440b101-26a7-49e8-a737-c1b65eb56093","Type":"ContainerStarted","Data":"2eeeb25a47b8d6526e1b320cae3322a9caff40f9f9d0505eb9a35c6c7feddc5e"}
Sep 09 10:30:53.369908 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:53.369868    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cfb7697-sjhff" event={"ID":"525c4513-e445-4135-9766-9bd1ef103e8a","Type":"ContainerStarted","Data":"65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5"}
Sep 09 10:30:53.370048 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:53.369933    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cfb7697-sjhff" event={"ID":"525c4513-e445-4135-9766-9bd1ef103e8a","Type":"ContainerStarted","Data":"6a99e2d0ad93a9a107c842b54f17f9e339f274eb0c25d6f0be99f463d371c1c4"}
Sep 09 10:30:53.371431 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:53.371401    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d" event={"ID":"9848bce8-6ce5-47a3-839d-ebd153b2ed07","Type":"ContainerStarted","Data":"00140a2440cc83f38bb2df23e53c209010cfe8a466aab0f274c3ecb6d4ec20fb"}
Sep 09 10:30:53.373699 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:53.373677    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn" event={"ID":"c688af9f-4254-46ce-89c1-732ca4651855","Type":"ContainerStarted","Data":"753c19fc38c464702626147990e8d6d038b826146a734aaca995d58f809bfd9d"}
Sep 09 10:30:53.373779 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:53.373731    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn" event={"ID":"c688af9f-4254-46ce-89c1-732ca4651855","Type":"ContainerStarted","Data":"f3cb9266330d8b8d8baca977780788bffb57307acc9296e6017c812381a97a96"}
Sep 09 10:30:53.373779 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:53.373757    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn" event={"ID":"c688af9f-4254-46ce-89c1-732ca4651855","Type":"ContainerStarted","Data":"322796a88e57462e23c4a075be6ccdc11ab84eb6553f0728e67485c3e323fae2"}
Sep 09 10:30:53.398639 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:53.398592    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66cfb7697-sjhff" podStartSLOduration=2.398576987 podStartE2EDuration="2.398576987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:53.396662374 +0000 UTC m=+295.452221426" watchObservedRunningTime="2026-09-09 10:30:53.398576987 +0000 UTC m=+295.454136037"
Sep 09 10:30:54.382172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:54.382130    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d" event={"ID":"9848bce8-6ce5-47a3-839d-ebd153b2ed07","Type":"ContainerStarted","Data":"271f9d5071ee9edaf8cf96b29d8e61f89d04ba5b0148f0e68e77141a8ea9a818"}
Sep 09 10:30:54.400351 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:54.400299    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d" podStartSLOduration=1.909283835 podStartE2EDuration="3.400273198s" totalImagesPullingTime="1.490989363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:54.398981718 +0000 UTC m=+296.454540768" watchObservedRunningTime="2026-09-09 10:30:54.400273198 +0000 UTC m=+296.455832263"
Sep 09 10:30:55.330154 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.330128    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4c2kf"
Sep 09 10:30:55.388500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.388454    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8" event={"ID":"d440b101-26a7-49e8-a737-c1b65eb56093","Type":"ContainerStarted","Data":"24ecc2e89ef8f267129a7e65b4e4d53319d3dfdef496fdbf92ee9c1dbd3c4ecf"}
Sep 09 10:30:55.388849 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.388689    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:55.391078 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.391046    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerStarted","Data":"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508"}
Sep 09 10:30:55.394406 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.394385    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8"
Sep 09 10:30:55.394936 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.394912    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn" event={"ID":"c688af9f-4254-46ce-89c1-732ca4651855","Type":"ContainerStarted","Data":"8790563c6725403c0344714c7b4540d4ba5a504eb8f38958e4523f4a7bf1c76e"}
Sep 09 10:30:55.395041 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.394975    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn" event={"ID":"c688af9f-4254-46ce-89c1-732ca4651855","Type":"ContainerStarted","Data":"59f0342f0d457e778d1b59d003b5d4e60f1fb639259e9113d7630cacbb2ccb93"}
Sep 09 10:30:55.395041 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.395016    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn" event={"ID":"c688af9f-4254-46ce-89c1-732ca4651855","Type":"ContainerStarted","Data":"4135e43c3044bc8cbb1404d6e7ae374448af0312447fef6c9ed259590d92dbc1"}
Sep 09 10:30:55.405488 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.405450    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vhwp8" podStartSLOduration=2.521051714 podStartE2EDuration="4.405436838s" totalImagesPullingTime="1.884385124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:55.404299051 +0000 UTC m=+297.459858092" watchObservedRunningTime="2026-09-09 10:30:55.405436838 +0000 UTC m=+297.460995911"
Sep 09 10:30:55.430760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:55.430717    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn" podStartSLOduration=1.668594725 podStartE2EDuration="5.430700716s" totalImagesPullingTime="3.762105991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:55.430004067 +0000 UTC m=+297.485563119" watchObservedRunningTime="2026-09-09 10:30:55.430700716 +0000 UTC m=+297.486259768"
Sep 09 10:30:56.402721 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:56.402674    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerStarted","Data":"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8"}
Sep 09 10:30:56.403118 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:56.402791    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerStarted","Data":"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6"}
Sep 09 10:30:56.403202 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:56.403184    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:56.403271 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:56.403255    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerStarted","Data":"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb"}
Sep 09 10:30:56.403348 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:56.403312    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerStarted","Data":"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c"}
Sep 09 10:30:56.403401 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:56.403357    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerStarted","Data":"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9"}
Sep 09 10:30:56.431912 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:56.431834    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.473085327 podStartE2EDuration="8.431811614s" totalImagesPullingTime="4.944004491s" totalInitContainerRuntime="14.721796ms" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:56.431531189 +0000 UTC m=+298.487090327" watchObservedRunningTime="2026-09-09 10:30:56.431811614 +0000 UTC m=+298.487370666"
Sep 09 10:30:57.414446 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:57.414419    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b975d59c4-qp5jn"
Sep 09 10:30:57.704285 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:57.704211    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b656c84bb-9hx2l"]
Sep 09 10:30:57.708814 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:57.708779    2586 patch_prober.go:28] interesting pod/image-registry-5b656c84bb-9hx2l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 10:30:57.708969 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:57.708837    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" podUID="775fc55b-45cc-4506-b06e-822381628930" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:30:58.390332 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:30:58.390178    2586 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 10:31:00.469152 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:00.469121    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:31:02.218025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:02.217987    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:31:02.218453 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:02.218075    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:31:02.223013 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:02.222989    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:31:02.431783 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:02.431757    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:31:07.709058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:07.709029    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:31:11.716931 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:11.716882    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:31:11.717311 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:11.716957    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:31:16.873429 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:16.873382    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7597dccf8d-4lcwk" podUID="b210a91c-3977-4e96-8688-6f3a4d6a291c" containerName="console" containerID="cri-o://aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09" gracePeriod=15
Sep 09 10:31:17.108686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.108658    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:31:17.231129 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231051    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-oauth-config\") pod \"b210a91c-3977-4e96-8688-6f3a4d6a291c\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") "
Sep 09 10:31:17.231129 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231114    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-config\") pod \"b210a91c-3977-4e96-8688-6f3a4d6a291c\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") "
Sep 09 10:31:17.231129 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231141    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-serving-cert\") pod \"b210a91c-3977-4e96-8688-6f3a4d6a291c\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") "
Sep 09 10:31:17.231447 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231158    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r7xm\" (UniqueName: \"kubernetes.io/projected/b210a91c-3977-4e96-8688-6f3a4d6a291c-kube-api-access-5r7xm\") pod \"b210a91c-3977-4e96-8688-6f3a4d6a291c\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") "
Sep 09 10:31:17.231447 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231179    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-service-ca\") pod \"b210a91c-3977-4e96-8688-6f3a4d6a291c\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") "
Sep 09 10:31:17.231447 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231196    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-oauth-serving-cert\") pod \"b210a91c-3977-4e96-8688-6f3a4d6a291c\" (UID: \"b210a91c-3977-4e96-8688-6f3a4d6a291c\") "
Sep 09 10:31:17.231594 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231563    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-config" (OuterVolumeSpecName: "console-config") pod "b210a91c-3977-4e96-8688-6f3a4d6a291c" (UID: "b210a91c-3977-4e96-8688-6f3a4d6a291c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:17.231767 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231724    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-service-ca" (OuterVolumeSpecName: "service-ca") pod "b210a91c-3977-4e96-8688-6f3a4d6a291c" (UID: "b210a91c-3977-4e96-8688-6f3a4d6a291c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:17.231767 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.231742    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b210a91c-3977-4e96-8688-6f3a4d6a291c" (UID: "b210a91c-3977-4e96-8688-6f3a4d6a291c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:17.233493 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.233466    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b210a91c-3977-4e96-8688-6f3a4d6a291c-kube-api-access-5r7xm" (OuterVolumeSpecName: "kube-api-access-5r7xm") pod "b210a91c-3977-4e96-8688-6f3a4d6a291c" (UID: "b210a91c-3977-4e96-8688-6f3a4d6a291c"). InnerVolumeSpecName "kube-api-access-5r7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:31:17.233580 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.233489    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b210a91c-3977-4e96-8688-6f3a4d6a291c" (UID: "b210a91c-3977-4e96-8688-6f3a4d6a291c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:31:17.233580 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.233553    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b210a91c-3977-4e96-8688-6f3a4d6a291c" (UID: "b210a91c-3977-4e96-8688-6f3a4d6a291c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:31:17.332471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.332423    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:17.332471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.332467    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:17.332471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.332477    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b210a91c-3977-4e96-8688-6f3a4d6a291c-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:17.332471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.332486    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r7xm\" (UniqueName: \"kubernetes.io/projected/b210a91c-3977-4e96-8688-6f3a4d6a291c-kube-api-access-5r7xm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:17.332698 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.332495    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:17.332698 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.332504    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b210a91c-3977-4e96-8688-6f3a4d6a291c-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:17.486586 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.486500    2586 generic.go:356] "Generic (PLEG): container finished" podID="b210a91c-3977-4e96-8688-6f3a4d6a291c" containerID="aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09" exitCode=0
Sep 09 10:31:17.486586 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.486557    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7597dccf8d-4lcwk"
Sep 09 10:31:17.486801 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.486594    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7597dccf8d-4lcwk" event={"ID":"b210a91c-3977-4e96-8688-6f3a4d6a291c","Type":"ContainerDied","Data":"aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09"}
Sep 09 10:31:17.486801 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.486646    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7597dccf8d-4lcwk" event={"ID":"b210a91c-3977-4e96-8688-6f3a4d6a291c","Type":"ContainerDied","Data":"a2fd3c435c366e03848d0a28e37a86b6389517047148e48cd829f9ec54d3273f"}
Sep 09 10:31:17.486801 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.486669    2586 scope.go:117] "RemoveContainer" containerID="aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09"
Sep 09 10:31:17.497244 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.497222    2586 scope.go:117] "RemoveContainer" containerID="aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09"
Sep 09 10:31:17.497519 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:31:17.497496    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09\": container with ID starting with aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09 not found: ID does not exist" containerID="aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09"
Sep 09 10:31:17.497593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.497525    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09"} err="failed to get container status \"aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09\": rpc error: code = NotFound desc = could not find container \"aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09\": container with ID starting with aa2b64747773f8740418f1fbc65f73f3e31b707f0ebd919c67b9f22ecfbf2f09 not found: ID does not exist"
Sep 09 10:31:17.507133 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.507108    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7597dccf8d-4lcwk"]
Sep 09 10:31:17.510768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:17.510748    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7597dccf8d-4lcwk"]
Sep 09 10:31:18.474133 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:18.474102    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b210a91c-3977-4e96-8688-6f3a4d6a291c" path="/var/lib/kubelet/pods/b210a91c-3977-4e96-8688-6f3a4d6a291c/volumes"
Sep 09 10:31:21.504000 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:21.503914    2586 generic.go:356] "Generic (PLEG): container finished" podID="db6c7c40-96e8-473e-ae77-35168a674dbe" containerID="08eb3c09e327ceefefee95cbc8b67bb7c4060d8a984afcd12aa7c3be1b26f68e" exitCode=0
Sep 09 10:31:21.504000 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:21.503990    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz" event={"ID":"db6c7c40-96e8-473e-ae77-35168a674dbe","Type":"ContainerDied","Data":"08eb3c09e327ceefefee95cbc8b67bb7c4060d8a984afcd12aa7c3be1b26f68e"}
Sep 09 10:31:21.504381 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:21.504360    2586 scope.go:117] "RemoveContainer" containerID="08eb3c09e327ceefefee95cbc8b67bb7c4060d8a984afcd12aa7c3be1b26f68e"
Sep 09 10:31:22.509724 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:22.509691    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-rmszz" event={"ID":"db6c7c40-96e8-473e-ae77-35168a674dbe","Type":"ContainerStarted","Data":"0d33e53c25abe09a510f1c5a0c5763e143dd2e34eda817321c7e733efcd1a032"}
Sep 09 10:31:22.724180 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:22.724141    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" podUID="775fc55b-45cc-4506-b06e-822381628930" containerName="registry" containerID="cri-o://21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6" gracePeriod=30
Sep 09 10:31:22.967500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:22.967478    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:31:23.086461 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086430    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-image-registry-private-configuration\") pod \"775fc55b-45cc-4506-b06e-822381628930\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") "
Sep 09 10:31:23.086628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086495    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-installation-pull-secrets\") pod \"775fc55b-45cc-4506-b06e-822381628930\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") "
Sep 09 10:31:23.086628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086538    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-trusted-ca\") pod \"775fc55b-45cc-4506-b06e-822381628930\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") "
Sep 09 10:31:23.086628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086555    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-bound-sa-token\") pod \"775fc55b-45cc-4506-b06e-822381628930\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") "
Sep 09 10:31:23.086628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086569    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") pod \"775fc55b-45cc-4506-b06e-822381628930\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") "
Sep 09 10:31:23.086628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086588    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5fpq\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-kube-api-access-l5fpq\") pod \"775fc55b-45cc-4506-b06e-822381628930\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") "
Sep 09 10:31:23.086875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086639    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/775fc55b-45cc-4506-b06e-822381628930-ca-trust-extracted\") pod \"775fc55b-45cc-4506-b06e-822381628930\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") "
Sep 09 10:31:23.086875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086670    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-registry-certificates\") pod \"775fc55b-45cc-4506-b06e-822381628930\" (UID: \"775fc55b-45cc-4506-b06e-822381628930\") "
Sep 09 10:31:23.087027 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.086986    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "775fc55b-45cc-4506-b06e-822381628930" (UID: "775fc55b-45cc-4506-b06e-822381628930"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:23.087359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.087301    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "775fc55b-45cc-4506-b06e-822381628930" (UID: "775fc55b-45cc-4506-b06e-822381628930"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:31:23.089147 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.089105    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-kube-api-access-l5fpq" (OuterVolumeSpecName: "kube-api-access-l5fpq") pod "775fc55b-45cc-4506-b06e-822381628930" (UID: "775fc55b-45cc-4506-b06e-822381628930"). InnerVolumeSpecName "kube-api-access-l5fpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:31:23.089284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.089255    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "775fc55b-45cc-4506-b06e-822381628930" (UID: "775fc55b-45cc-4506-b06e-822381628930"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:31:23.089393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.089334    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "775fc55b-45cc-4506-b06e-822381628930" (UID: "775fc55b-45cc-4506-b06e-822381628930"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:31:23.089451 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.089391    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "775fc55b-45cc-4506-b06e-822381628930" (UID: "775fc55b-45cc-4506-b06e-822381628930"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:31:23.089451 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.089404    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "775fc55b-45cc-4506-b06e-822381628930" (UID: "775fc55b-45cc-4506-b06e-822381628930"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:31:23.103778 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.103747    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775fc55b-45cc-4506-b06e-822381628930-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "775fc55b-45cc-4506-b06e-822381628930" (UID: "775fc55b-45cc-4506-b06e-822381628930"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:31:23.188188 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.188153    2586 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-installation-pull-secrets\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:23.188188 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.188183    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-trusted-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:23.188188 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.188192    2586 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-bound-sa-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:23.188188 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.188200    2586 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-registry-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:23.188432 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.188209    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5fpq\" (UniqueName: \"kubernetes.io/projected/775fc55b-45cc-4506-b06e-822381628930-kube-api-access-l5fpq\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:23.188432 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.188217    2586 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/775fc55b-45cc-4506-b06e-822381628930-ca-trust-extracted\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:23.188432 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.188224    2586 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/775fc55b-45cc-4506-b06e-822381628930-registry-certificates\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:23.188432 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.188233    2586 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/775fc55b-45cc-4506-b06e-822381628930-image-registry-private-configuration\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:31:23.515164 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.515131    2586 generic.go:356] "Generic (PLEG): container finished" podID="775fc55b-45cc-4506-b06e-822381628930" containerID="21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6" exitCode=0
Sep 09 10:31:23.515542 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.515170    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l"
Sep 09 10:31:23.515542 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.515221    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" event={"ID":"775fc55b-45cc-4506-b06e-822381628930","Type":"ContainerDied","Data":"21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6"}
Sep 09 10:31:23.515542 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.515269    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b656c84bb-9hx2l" event={"ID":"775fc55b-45cc-4506-b06e-822381628930","Type":"ContainerDied","Data":"03e800e5866127b23c23b130409d7386ab0b8fc3c1e51e491d8304eccc339315"}
Sep 09 10:31:23.515542 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.515292    2586 scope.go:117] "RemoveContainer" containerID="21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6"
Sep 09 10:31:23.524328 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.524311    2586 scope.go:117] "RemoveContainer" containerID="21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6"
Sep 09 10:31:23.524564 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:31:23.524548    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6\": container with ID starting with 21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6 not found: ID does not exist" containerID="21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6"
Sep 09 10:31:23.524615 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.524571    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6"} err="failed to get container status \"21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6\": rpc error: code = NotFound desc = could not find container \"21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6\": container with ID starting with 21672ceebb7718696eedc957372108324e3d74320521c1d264b6569d40219cb6 not found: ID does not exist"
Sep 09 10:31:23.544059 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.544039    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b656c84bb-9hx2l"]
Sep 09 10:31:23.548351 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:23.548330    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b656c84bb-9hx2l"]
Sep 09 10:31:24.473786 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:24.473750    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775fc55b-45cc-4506-b06e-822381628930" path="/var/lib/kubelet/pods/775fc55b-45cc-4506-b06e-822381628930/volumes"
Sep 09 10:31:26.528700 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:26.528664    2586 generic.go:356] "Generic (PLEG): container finished" podID="572133a3-bac1-458a-b2ed-fb140a33bbe3" containerID="1de0e8264c0c4e276d6a12657e52d3ae9f12a56ab2153cbbfb7ac80d12c88db8" exitCode=0
Sep 09 10:31:26.529084 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:26.528706    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6" event={"ID":"572133a3-bac1-458a-b2ed-fb140a33bbe3","Type":"ContainerDied","Data":"1de0e8264c0c4e276d6a12657e52d3ae9f12a56ab2153cbbfb7ac80d12c88db8"}
Sep 09 10:31:26.529084 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:26.529032    2586 scope.go:117] "RemoveContainer" containerID="1de0e8264c0c4e276d6a12657e52d3ae9f12a56ab2153cbbfb7ac80d12c88db8"
Sep 09 10:31:27.535256 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:27.535224    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7bkz6" event={"ID":"572133a3-bac1-458a-b2ed-fb140a33bbe3","Type":"ContainerStarted","Data":"90d25c60e54175ee8e9a24728db47a482faab50a1400193654e18af39dc6c070"}
Sep 09 10:31:31.721955 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:31.721919    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:31:31.725721 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:31:31.725695    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-588d844b79-6xj8d"
Sep 09 10:32:07.361112 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.361076    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:32:07.361502 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.361363    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="alertmanager" containerID="cri-o://96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508" gracePeriod=120
Sep 09 10:32:07.361502 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.361474    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="config-reloader" containerID="cri-o://203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9" gracePeriod=120
Sep 09 10:32:07.361609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.361506    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="prom-label-proxy" containerID="cri-o://d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8" gracePeriod=120
Sep 09 10:32:07.361609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.361515    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy" containerID="cri-o://4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb" gracePeriod=120
Sep 09 10:32:07.361609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.361456    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy-web" containerID="cri-o://2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c" gracePeriod=120
Sep 09 10:32:07.361744 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.361449    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy-metric" containerID="cri-o://3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6" gracePeriod=120
Sep 09 10:32:07.698355 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.698277    2586 generic.go:356] "Generic (PLEG): container finished" podID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerID="d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8" exitCode=0
Sep 09 10:32:07.698355 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.698304    2586 generic.go:356] "Generic (PLEG): container finished" podID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerID="4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb" exitCode=0
Sep 09 10:32:07.698355 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.698313    2586 generic.go:356] "Generic (PLEG): container finished" podID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerID="203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9" exitCode=0
Sep 09 10:32:07.698355 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.698318    2586 generic.go:356] "Generic (PLEG): container finished" podID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerID="96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508" exitCode=0
Sep 09 10:32:07.698355 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.698347    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerDied","Data":"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8"}
Sep 09 10:32:07.698614 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.698408    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerDied","Data":"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb"}
Sep 09 10:32:07.698614 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.698443    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerDied","Data":"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9"}
Sep 09 10:32:07.698614 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:07.698473    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerDied","Data":"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508"}
Sep 09 10:32:08.607107 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.607073    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:08.694361 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694262    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-web\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694361 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694326    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-trusted-ca-bundle\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694361 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694352    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-volume\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694388    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694425    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-tls-assets\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694574    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-main-db\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694804 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694643    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-web-config\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694804 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694689    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-metrics-client-ca\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694804 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694730    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-cluster-tls-config\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694804 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694760    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-metric\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.694804 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694791    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.695081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694847    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qm9m\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-kube-api-access-6qm9m\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.695081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694864    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:08.695081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.694910    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-out\") pod \"60f808ac-2802-4c98-92b9-8a41c7ebe481\" (UID: \"60f808ac-2802-4c98-92b9-8a41c7ebe481\") "
Sep 09 10:32:08.695229 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.695177    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:32:08.695229 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.695190    2586 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.695331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.695300    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:08.697791 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.697757    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:08.697958 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.697818    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:08.698056 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.697971    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-volume" (OuterVolumeSpecName: "config-volume") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:08.698134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.698112    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-out" (OuterVolumeSpecName: "config-out") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:32:08.699384 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.699352    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-kube-api-access-6qm9m" (OuterVolumeSpecName: "kube-api-access-6qm9m") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "kube-api-access-6qm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:32:08.699384 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.699350    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:08.699631 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.699605    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:08.699631 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.699614    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:32:08.703319 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.703251    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:08.708007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.707977    2586 generic.go:356] "Generic (PLEG): container finished" podID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerID="3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6" exitCode=0
Sep 09 10:32:08.708007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.708004    2586 generic.go:356] "Generic (PLEG): container finished" podID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerID="2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c" exitCode=0
Sep 09 10:32:08.708147 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.708053    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerDied","Data":"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6"}
Sep 09 10:32:08.708147 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.708070    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:08.708147 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.708142    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerDied","Data":"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c"}
Sep 09 10:32:08.708310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.708172    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f808ac-2802-4c98-92b9-8a41c7ebe481","Type":"ContainerDied","Data":"1a39dade563c2aca2746dfc1eec6473dd5690b078c7f24a2e5c64f8b0f5a3056"}
Sep 09 10:32:08.708310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.708203    2586 scope.go:117] "RemoveContainer" containerID="d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8"
Sep 09 10:32:08.711187 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.711165    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-web-config" (OuterVolumeSpecName: "web-config") pod "60f808ac-2802-4c98-92b9-8a41c7ebe481" (UID: "60f808ac-2802-4c98-92b9-8a41c7ebe481"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:08.715429 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.715406    2586 scope.go:117] "RemoveContainer" containerID="3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6"
Sep 09 10:32:08.722232 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.722215    2586 scope.go:117] "RemoveContainer" containerID="4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb"
Sep 09 10:32:08.728951 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.728933    2586 scope.go:117] "RemoveContainer" containerID="2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c"
Sep 09 10:32:08.735630 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.735613    2586 scope.go:117] "RemoveContainer" containerID="203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9"
Sep 09 10:32:08.742644 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.742622    2586 scope.go:117] "RemoveContainer" containerID="96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508"
Sep 09 10:32:08.749676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.749654    2586 scope.go:117] "RemoveContainer" containerID="8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e"
Sep 09 10:32:08.756476 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.756456    2586 scope.go:117] "RemoveContainer" containerID="d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8"
Sep 09 10:32:08.756755 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:32:08.756738    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8\": container with ID starting with d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8 not found: ID does not exist" containerID="d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8"
Sep 09 10:32:08.756821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.756768    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8"} err="failed to get container status \"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8\": rpc error: code = NotFound desc = could not find container \"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8\": container with ID starting with d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8 not found: ID does not exist"
Sep 09 10:32:08.756821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.756796    2586 scope.go:117] "RemoveContainer" containerID="3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6"
Sep 09 10:32:08.757188 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:32:08.757165    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6\": container with ID starting with 3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6 not found: ID does not exist" containerID="3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6"
Sep 09 10:32:08.757231 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.757195    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6"} err="failed to get container status \"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6\": rpc error: code = NotFound desc = could not find container \"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6\": container with ID starting with 3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6 not found: ID does not exist"
Sep 09 10:32:08.757231 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.757214    2586 scope.go:117] "RemoveContainer" containerID="4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb"
Sep 09 10:32:08.757472 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:32:08.757457    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb\": container with ID starting with 4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb not found: ID does not exist" containerID="4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb"
Sep 09 10:32:08.757515 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.757476    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb"} err="failed to get container status \"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb\": rpc error: code = NotFound desc = could not find container \"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb\": container with ID starting with 4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb not found: ID does not exist"
Sep 09 10:32:08.757515 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.757490    2586 scope.go:117] "RemoveContainer" containerID="2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c"
Sep 09 10:32:08.757745 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:32:08.757724    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c\": container with ID starting with 2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c not found: ID does not exist" containerID="2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c"
Sep 09 10:32:08.757803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.757750    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c"} err="failed to get container status \"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c\": rpc error: code = NotFound desc = could not find container \"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c\": container with ID starting with 2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c not found: ID does not exist"
Sep 09 10:32:08.757803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.757765    2586 scope.go:117] "RemoveContainer" containerID="203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9"
Sep 09 10:32:08.758044 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:32:08.758018    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9\": container with ID starting with 203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9 not found: ID does not exist" containerID="203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9"
Sep 09 10:32:08.758095 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.758050    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9"} err="failed to get container status \"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9\": rpc error: code = NotFound desc = could not find container \"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9\": container with ID starting with 203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9 not found: ID does not exist"
Sep 09 10:32:08.758095 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.758065    2586 scope.go:117] "RemoveContainer" containerID="96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508"
Sep 09 10:32:08.758321 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:32:08.758304    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508\": container with ID starting with 96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508 not found: ID does not exist" containerID="96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508"
Sep 09 10:32:08.758376 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.758329    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508"} err="failed to get container status \"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508\": rpc error: code = NotFound desc = could not find container \"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508\": container with ID starting with 96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508 not found: ID does not exist"
Sep 09 10:32:08.758376 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.758352    2586 scope.go:117] "RemoveContainer" containerID="8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e"
Sep 09 10:32:08.758600 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:32:08.758583    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e\": container with ID starting with 8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e not found: ID does not exist" containerID="8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e"
Sep 09 10:32:08.758638 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.758610    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e"} err="failed to get container status \"8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e\": rpc error: code = NotFound desc = could not find container \"8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e\": container with ID starting with 8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e not found: ID does not exist"
Sep 09 10:32:08.758638 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.758626    2586 scope.go:117] "RemoveContainer" containerID="d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8"
Sep 09 10:32:08.758843 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.758824    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8"} err="failed to get container status \"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8\": rpc error: code = NotFound desc = could not find container \"d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8\": container with ID starting with d3c516d4d7dff77e274dfedfc9133e36af8f01b9e269a286f4e12b6406f6a0d8 not found: ID does not exist"
Sep 09 10:32:08.758922 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.758844    2586 scope.go:117] "RemoveContainer" containerID="3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6"
Sep 09 10:32:08.759078 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.759061    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6"} err="failed to get container status \"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6\": rpc error: code = NotFound desc = could not find container \"3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6\": container with ID starting with 3eb499f3cf3025b21a0b4460d2aecc2519d5871cfdd2132e8ddf9af12ade37f6 not found: ID does not exist"
Sep 09 10:32:08.759119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.759078    2586 scope.go:117] "RemoveContainer" containerID="4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb"
Sep 09 10:32:08.759323 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.759306    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb"} err="failed to get container status \"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb\": rpc error: code = NotFound desc = could not find container \"4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb\": container with ID starting with 4f4fa5038fd848bb2267cfe490abd84044269c82dc6a3a8c1846c20dd8ab6bbb not found: ID does not exist"
Sep 09 10:32:08.759362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.759323    2586 scope.go:117] "RemoveContainer" containerID="2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c"
Sep 09 10:32:08.759566 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.759550    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c"} err="failed to get container status \"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c\": rpc error: code = NotFound desc = could not find container \"2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c\": container with ID starting with 2a361ef548655d1e236021f503791d136c328d777deaf7392c16890840b2bf0c not found: ID does not exist"
Sep 09 10:32:08.759631 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.759568    2586 scope.go:117] "RemoveContainer" containerID="203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9"
Sep 09 10:32:08.759849 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.759831    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9"} err="failed to get container status \"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9\": rpc error: code = NotFound desc = could not find container \"203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9\": container with ID starting with 203e3da56f7b089e79a480ae8bd706ffef848813cb697553f63b28f976fa9cd9 not found: ID does not exist"
Sep 09 10:32:08.759921 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.759849    2586 scope.go:117] "RemoveContainer" containerID="96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508"
Sep 09 10:32:08.760146 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.760131    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508"} err="failed to get container status \"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508\": rpc error: code = NotFound desc = could not find container \"96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508\": container with ID starting with 96a32e600d126d6f5bbd50eb8fdebc558edf1706c0d5f5a976ce83ceebc5a508 not found: ID does not exist"
Sep 09 10:32:08.760187 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.760147    2586 scope.go:117] "RemoveContainer" containerID="8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e"
Sep 09 10:32:08.760380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.760361    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e"} err="failed to get container status \"8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e\": rpc error: code = NotFound desc = could not find container \"8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e\": container with ID starting with 8beba2452863b20992989b100bba485486f6e047e2580ade8b1ba79225b0867e not found: ID does not exist"
Sep 09 10:32:08.796002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.795975    2586 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-tls-assets\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.795998    2586 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-alertmanager-main-db\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796009    2586 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-web-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796019    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f808ac-2802-4c98-92b9-8a41c7ebe481-metrics-client-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796029    2586 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-cluster-tls-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796038    2586 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796048    2586 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796056    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qm9m\" (UniqueName: \"kubernetes.io/projected/60f808ac-2802-4c98-92b9-8a41c7ebe481-kube-api-access-6qm9m\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796064    2586 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-out\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796072    2586 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796080    2586 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-config-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:08.796108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:08.796089    2586 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f808ac-2802-4c98-92b9-8a41c7ebe481-secret-alertmanager-main-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:09.033138 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.033102    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:32:09.037820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.037795    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:32:09.066713 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.066683    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:32:09.067061 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067044    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775fc55b-45cc-4506-b06e-822381628930" containerName="registry"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067063    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="775fc55b-45cc-4506-b06e-822381628930" containerName="registry"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067079    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy-web"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067087    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy-web"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067103    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy-metric"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067111    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy-metric"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067125    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="init-config-reloader"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067132    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="init-config-reloader"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067141    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="alertmanager"
Sep 09 10:32:09.067158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067149    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="alertmanager"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067162    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b210a91c-3977-4e96-8688-6f3a4d6a291c" containerName="console"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067170    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b210a91c-3977-4e96-8688-6f3a4d6a291c" containerName="console"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067183    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="config-reloader"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067193    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="config-reloader"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067203    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="prom-label-proxy"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067211    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="prom-label-proxy"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067232    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067240    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067315    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy-web"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067328    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b210a91c-3977-4e96-8688-6f3a4d6a291c" containerName="console"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067336    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="alertmanager"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067348    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="775fc55b-45cc-4506-b06e-822381628930" containerName="registry"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067356    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="config-reloader"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067365    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy-metric"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067373    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="prom-label-proxy"
Sep 09 10:32:09.067575 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.067382    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" containerName="kube-rbac-proxy"
Sep 09 10:32:09.072748 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.072727    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.074926 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.074902    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 10:32:09.075179 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.075162    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 10:32:09.075265 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.075181    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 10:32:09.075265 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.075162    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 10:32:09.075505 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.075490    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 10:32:09.075577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.075505    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 10:32:09.075679 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.075658    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 10:32:09.075737 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.075698    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nhjq2\""
Sep 09 10:32:09.075905 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.075880    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 10:32:09.082020 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.082003    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 10:32:09.093549 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.093530    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:32:09.199824 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.199787    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.199824 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.199824    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/678bf4b7-00cc-4db5-afa4-1eefb60c8016-config-out\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.199854    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-web-config\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.199931    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678bf4b7-00cc-4db5-afa4-1eefb60c8016-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.199984    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.200008    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678bf4b7-00cc-4db5-afa4-1eefb60c8016-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.200036    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-config-volume\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200219 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.200051    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200219 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.200072    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/678bf4b7-00cc-4db5-afa4-1eefb60c8016-tls-assets\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200219 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.200096    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200219 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.200142    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65ns\" (UniqueName: \"kubernetes.io/projected/678bf4b7-00cc-4db5-afa4-1eefb60c8016-kube-api-access-x65ns\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200219 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.200201    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.200395 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.200237    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/678bf4b7-00cc-4db5-afa4-1eefb60c8016-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301616 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301540    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/678bf4b7-00cc-4db5-afa4-1eefb60c8016-tls-assets\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301616 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301572    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301616 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301593    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65ns\" (UniqueName: \"kubernetes.io/projected/678bf4b7-00cc-4db5-afa4-1eefb60c8016-kube-api-access-x65ns\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301634    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301712    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/678bf4b7-00cc-4db5-afa4-1eefb60c8016-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301764    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301790    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/678bf4b7-00cc-4db5-afa4-1eefb60c8016-config-out\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301817    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-web-config\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.301875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678bf4b7-00cc-4db5-afa4-1eefb60c8016-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.302191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301933    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.302191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.301969    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678bf4b7-00cc-4db5-afa4-1eefb60c8016-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.302191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.302004    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-config-volume\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.302191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.302028    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.302191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.302148    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/678bf4b7-00cc-4db5-afa4-1eefb60c8016-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.303343 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.303315    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678bf4b7-00cc-4db5-afa4-1eefb60c8016-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.304645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.304513    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/678bf4b7-00cc-4db5-afa4-1eefb60c8016-config-out\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.304761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.304741    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.304823 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.304793    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.304870 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.304851    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/678bf4b7-00cc-4db5-afa4-1eefb60c8016-tls-assets\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.305059 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.305039    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.305122 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.305075    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.305122 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.305088    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.305303 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.305283    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678bf4b7-00cc-4db5-afa4-1eefb60c8016-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.306115 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.306095    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-config-volume\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.306632 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.306615    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/678bf4b7-00cc-4db5-afa4-1eefb60c8016-web-config\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.310859 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.310843    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65ns\" (UniqueName: \"kubernetes.io/projected/678bf4b7-00cc-4db5-afa4-1eefb60c8016-kube-api-access-x65ns\") pod \"alertmanager-main-0\" (UID: \"678bf4b7-00cc-4db5-afa4-1eefb60c8016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.382933 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.382907    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 10:32:09.661930 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.661887    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 10:32:09.667828 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:32:09.667801    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678bf4b7_00cc_4db5_afa4_1eefb60c8016.slice/crio-5576d92d273d064633a02161f2f257ab35578738348fdc25ab164b0769b3624b WatchSource:0}: Error finding container 5576d92d273d064633a02161f2f257ab35578738348fdc25ab164b0769b3624b: Status 404 returned error can't find the container with id 5576d92d273d064633a02161f2f257ab35578738348fdc25ab164b0769b3624b
Sep 09 10:32:09.713250 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:09.713219    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678bf4b7-00cc-4db5-afa4-1eefb60c8016","Type":"ContainerStarted","Data":"5576d92d273d064633a02161f2f257ab35578738348fdc25ab164b0769b3624b"}
Sep 09 10:32:10.477956 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:10.476000    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f808ac-2802-4c98-92b9-8a41c7ebe481" path="/var/lib/kubelet/pods/60f808ac-2802-4c98-92b9-8a41c7ebe481/volumes"
Sep 09 10:32:10.719363 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:10.719280    2586 generic.go:356] "Generic (PLEG): container finished" podID="678bf4b7-00cc-4db5-afa4-1eefb60c8016" containerID="8eae1bf4ef0137fefcc77d57367e1e5f1fa4c2a5912b473da4faf103072d0cee" exitCode=0
Sep 09 10:32:10.719363 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:10.719329    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678bf4b7-00cc-4db5-afa4-1eefb60c8016","Type":"ContainerDied","Data":"8eae1bf4ef0137fefcc77d57367e1e5f1fa4c2a5912b473da4faf103072d0cee"}
Sep 09 10:32:11.381525 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.381491    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"]
Sep 09 10:32:11.384960 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.384938    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.387144 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.387119    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 10:32:11.387238 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.387143    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 10:32:11.387238 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.387143    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 10:32:11.387334 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.387146    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 10:32:11.387334 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.387315    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 10:32:11.387465 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.387448    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bnnf9\""
Sep 09 10:32:11.393293 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.393270    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 10:32:11.409512 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.409489    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"]
Sep 09 10:32:11.520533 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.520505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpqw\" (UniqueName: \"kubernetes.io/projected/077d1c94-d212-467f-a555-182fcd143100-kube-api-access-tnpqw\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.520686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.520547    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-telemeter-client-tls\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.520686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.520583    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-metrics-client-ca\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.520686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.520610    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-federate-client-tls\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.520686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.520637    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.520686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.520674    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.520847 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.520701    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-secret-telemeter-client\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.520847 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.520719    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-serving-certs-ca-bundle\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.621700 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.621666    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-secret-telemeter-client\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.621700 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.621700    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-serving-certs-ca-bundle\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.621947 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.621846    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpqw\" (UniqueName: \"kubernetes.io/projected/077d1c94-d212-467f-a555-182fcd143100-kube-api-access-tnpqw\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.621947 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.621886    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-telemeter-client-tls\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.621947 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.621944    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-metrics-client-ca\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.622096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.621961    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-federate-client-tls\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.622096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.621988    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.622096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.622005    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.622514 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.622485    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-serving-certs-ca-bundle\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.622665 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.622627    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-metrics-client-ca\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.622844 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.622820    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/077d1c94-d212-467f-a555-182fcd143100-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.624281 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.624251    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-secret-telemeter-client\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.624559 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.624494    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.624628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.624583    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-federate-client-tls\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.624675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.624656    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/077d1c94-d212-467f-a555-182fcd143100-telemeter-client-tls\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.629740 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.629723    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpqw\" (UniqueName: \"kubernetes.io/projected/077d1c94-d212-467f-a555-182fcd143100-kube-api-access-tnpqw\") pod \"telemeter-client-7cc9b8d685-jjf8p\" (UID: \"077d1c94-d212-467f-a555-182fcd143100\") " pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.695765 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.695696    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"
Sep 09 10:32:11.729231 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.729198    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678bf4b7-00cc-4db5-afa4-1eefb60c8016","Type":"ContainerStarted","Data":"1f0b6525fdaae6fcdb377c26b1a92426c760b60ac67d24f73c5111d3a59cd7f5"}
Sep 09 10:32:11.729641 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.729280    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678bf4b7-00cc-4db5-afa4-1eefb60c8016","Type":"ContainerStarted","Data":"c7d7cdde5926237d793cf1a94bd6948a62128bc182e4edbf78b45e8132e2d32b"}
Sep 09 10:32:11.729641 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.729309    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678bf4b7-00cc-4db5-afa4-1eefb60c8016","Type":"ContainerStarted","Data":"783f87a444b6036aab5f8ce4f8bd5cb7a70bfb5f728110db173935f6a31bb4d2"}
Sep 09 10:32:11.729641 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.729341    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678bf4b7-00cc-4db5-afa4-1eefb60c8016","Type":"ContainerStarted","Data":"28c68c403e2a1edc72ddfb6a83d69c4b76810931fb10417e866c2dfe30b632c3"}
Sep 09 10:32:11.729641 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.729383    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678bf4b7-00cc-4db5-afa4-1eefb60c8016","Type":"ContainerStarted","Data":"36669bac90609dc7b242ff474302be7e32efbcae02b2f858f7c599c6c060a241"}
Sep 09 10:32:11.729641 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.729411    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678bf4b7-00cc-4db5-afa4-1eefb60c8016","Type":"ContainerStarted","Data":"7e41e4bc2dc1d6de442e2d43c0a149ddfd8fdd846e82ca7d9c4b3e8a32eea83e"}
Sep 09 10:32:11.756840 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.756787    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.73989261 podStartE2EDuration="2.756766783s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.874173ms" isStatefulPod=false podCreationTimestamp="2026-09-09 10:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:32:11.755388912 +0000 UTC m=+373.810948009" watchObservedRunningTime="2026-09-09 10:32:11.756766783 +0000 UTC m=+373.812325833"
Sep 09 10:32:11.988694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.988352    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p"]
Sep 09 10:32:11.990924 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:32:11.990879    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077d1c94_d212_467f_a555_182fcd143100.slice/crio-27e919290cc472c25f889175741efeb52fd3c4fe7bcb48ca302a439969bd1f41 WatchSource:0}: Error finding container 27e919290cc472c25f889175741efeb52fd3c4fe7bcb48ca302a439969bd1f41: Status 404 returned error can't find the container with id 27e919290cc472c25f889175741efeb52fd3c4fe7bcb48ca302a439969bd1f41
Sep 09 10:32:11.993067 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:11.993048    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:32:12.427738 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:12.427708    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:32:12.430061 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:12.430033    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d68f3e9e-ee75-4e19-afde-af8bcf7c76d6-metrics-certs\") pod \"network-metrics-daemon-d48rq\" (UID: \"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6\") " pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:32:12.472829 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:12.472797    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hq5pc\""
Sep 09 10:32:12.480620 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:12.480597    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d48rq"
Sep 09 10:32:12.735927 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:12.735842    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p" event={"ID":"077d1c94-d212-467f-a555-182fcd143100","Type":"ContainerStarted","Data":"27e919290cc472c25f889175741efeb52fd3c4fe7bcb48ca302a439969bd1f41"}
Sep 09 10:32:12.750448 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:12.750416    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d48rq"]
Sep 09 10:32:12.753152 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:32:12.753123    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68f3e9e_ee75_4e19_afde_af8bcf7c76d6.slice/crio-ea49e746982f04800e97270b9f5d9e2c90a0cb6e01dc59e8e10c5b2ea3796ce8 WatchSource:0}: Error finding container ea49e746982f04800e97270b9f5d9e2c90a0cb6e01dc59e8e10c5b2ea3796ce8: Status 404 returned error can't find the container with id ea49e746982f04800e97270b9f5d9e2c90a0cb6e01dc59e8e10c5b2ea3796ce8
Sep 09 10:32:13.742660 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:13.742617    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d48rq" event={"ID":"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6","Type":"ContainerStarted","Data":"ea49e746982f04800e97270b9f5d9e2c90a0cb6e01dc59e8e10c5b2ea3796ce8"}
Sep 09 10:32:14.751084 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:14.751051    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p" event={"ID":"077d1c94-d212-467f-a555-182fcd143100","Type":"ContainerStarted","Data":"1163115e52047af69751452903cfaf3a4d7522041102f426737f641a279d8136"}
Sep 09 10:32:14.751432 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:14.751128    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p" event={"ID":"077d1c94-d212-467f-a555-182fcd143100","Type":"ContainerStarted","Data":"5c79440c9d3b422e7db4be663340bd1b5fd916617786f67d0ab8628671b45420"}
Sep 09 10:32:14.751432 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:14.751160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p" event={"ID":"077d1c94-d212-467f-a555-182fcd143100","Type":"ContainerStarted","Data":"76dc6523fc9515b02a07431863579bce1682eb5d9553b1e5b170eb98187fb42c"}
Sep 09 10:32:14.777081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:14.777036    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7cc9b8d685-jjf8p" podStartSLOduration=1.933857041 podStartE2EDuration="3.777019156s" totalImagesPullingTime="1.843162115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:32:14.774819536 +0000 UTC m=+376.830378637" watchObservedRunningTime="2026-09-09 10:32:14.777019156 +0000 UTC m=+376.832578203"
Sep 09 10:32:14.874988 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:14.874960    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b7dcfd77-twktd"]
Sep 09 10:32:14.879340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:14.879323    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:14.888523 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:14.888499    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b7dcfd77-twktd"]
Sep 09 10:32:15.054246 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.054151    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrg9k\" (UniqueName: \"kubernetes.io/projected/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-kube-api-access-zrg9k\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.054246 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.054187    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-config\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.054246 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.054212    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-oauth-config\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.054459 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.054299    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-trusted-ca-bundle\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.054459 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.054322    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-service-ca\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.054459 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.054342    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-oauth-serving-cert\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.054459 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.054408    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-serving-cert\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.155148 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.155094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrg9k\" (UniqueName: \"kubernetes.io/projected/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-kube-api-access-zrg9k\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.155148 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.155157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-config\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.155379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.155192    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-oauth-config\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.155379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.155233    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-trusted-ca-bundle\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.155379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.155258    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-service-ca\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.155379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.155284    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-oauth-serving-cert\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.155379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.155328    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-serving-cert\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.156068 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.156042    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-service-ca\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.156180 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.156090    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-config\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.156222 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.156185    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-oauth-serving-cert\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.156297 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.156273    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-trusted-ca-bundle\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.157750 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.157720    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-serving-cert\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.157837 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.157782    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-oauth-config\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.165658 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.165639    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrg9k\" (UniqueName: \"kubernetes.io/projected/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-kube-api-access-zrg9k\") pod \"console-b7dcfd77-twktd\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") " pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.195748 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.195718    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:15.468991 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.468965    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b7dcfd77-twktd"]
Sep 09 10:32:15.470648 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:32:15.470623    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86b746e_dbc5_4e50_8a46_6ca5030b6d2e.slice/crio-96d8a0d06f4cd45fcaf2b6154ac1de9685e2a446fd3c5dbcff8f0b60d153f390 WatchSource:0}: Error finding container 96d8a0d06f4cd45fcaf2b6154ac1de9685e2a446fd3c5dbcff8f0b60d153f390: Status 404 returned error can't find the container with id 96d8a0d06f4cd45fcaf2b6154ac1de9685e2a446fd3c5dbcff8f0b60d153f390
Sep 09 10:32:15.757126 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.757032    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7dcfd77-twktd" event={"ID":"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e","Type":"ContainerStarted","Data":"6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a"}
Sep 09 10:32:15.757126 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.757082    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7dcfd77-twktd" event={"ID":"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e","Type":"ContainerStarted","Data":"96d8a0d06f4cd45fcaf2b6154ac1de9685e2a446fd3c5dbcff8f0b60d153f390"}
Sep 09 10:32:15.759020 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.758990    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d48rq" event={"ID":"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6","Type":"ContainerStarted","Data":"9ab0902e43e4d83904de68c148629e32e87dc113608c709ba27e989fdf414b3e"}
Sep 09 10:32:15.759133 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.759039    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d48rq" event={"ID":"d68f3e9e-ee75-4e19-afde-af8bcf7c76d6","Type":"ContainerStarted","Data":"de14c4ac9d93e7d2e05e49019c276ee3375b171d7f73966d97a697ec919d8ce2"}
Sep 09 10:32:15.781857 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.781816    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b7dcfd77-twktd" podStartSLOduration=1.781802247 podStartE2EDuration="1.781802247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:32:15.781444366 +0000 UTC m=+377.837003417" watchObservedRunningTime="2026-09-09 10:32:15.781802247 +0000 UTC m=+377.837361308"
Sep 09 10:32:15.798752 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:15.798708    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d48rq" podStartSLOduration=375.830100109 podStartE2EDuration="6m17.798692974s" totalImagesPullingTime="1.968592865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:32:15.797786038 +0000 UTC m=+377.853345092" watchObservedRunningTime="2026-09-09 10:32:15.798692974 +0000 UTC m=+377.854252024"
Sep 09 10:32:25.196298 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:25.196261    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:25.196700 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:25.196325    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:25.202003 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:25.201978    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:25.812169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:25.812139    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:32:25.864416 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:25.864376    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66cfb7697-sjhff"]
Sep 09 10:32:50.888111 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:50.888053    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66cfb7697-sjhff" podUID="525c4513-e445-4135-9766-9bd1ef103e8a" containerName="console" containerID="cri-o://65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5" gracePeriod=15
Sep 09 10:32:51.122848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.122822    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:32:51.265132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265052    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp5dl\" (UniqueName: \"kubernetes.io/projected/525c4513-e445-4135-9766-9bd1ef103e8a-kube-api-access-lp5dl\") pod \"525c4513-e445-4135-9766-9bd1ef103e8a\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") "
Sep 09 10:32:51.265132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265088    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-console-config\") pod \"525c4513-e445-4135-9766-9bd1ef103e8a\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") "
Sep 09 10:32:51.265132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265113    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-service-ca\") pod \"525c4513-e445-4135-9766-9bd1ef103e8a\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") "
Sep 09 10:32:51.265132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265130    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-oauth-config\") pod \"525c4513-e445-4135-9766-9bd1ef103e8a\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") "
Sep 09 10:32:51.265484 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265164    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-oauth-serving-cert\") pod \"525c4513-e445-4135-9766-9bd1ef103e8a\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") "
Sep 09 10:32:51.265484 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265225    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-trusted-ca-bundle\") pod \"525c4513-e445-4135-9766-9bd1ef103e8a\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") "
Sep 09 10:32:51.265484 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265256    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-serving-cert\") pod \"525c4513-e445-4135-9766-9bd1ef103e8a\" (UID: \"525c4513-e445-4135-9766-9bd1ef103e8a\") "
Sep 09 10:32:51.265650 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265571    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "525c4513-e445-4135-9766-9bd1ef103e8a" (UID: "525c4513-e445-4135-9766-9bd1ef103e8a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:51.265737 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265712    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "525c4513-e445-4135-9766-9bd1ef103e8a" (UID: "525c4513-e445-4135-9766-9bd1ef103e8a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:51.265795 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265753    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-service-ca" (OuterVolumeSpecName: "service-ca") pod "525c4513-e445-4135-9766-9bd1ef103e8a" (UID: "525c4513-e445-4135-9766-9bd1ef103e8a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:51.265858 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.265809    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-console-config" (OuterVolumeSpecName: "console-config") pod "525c4513-e445-4135-9766-9bd1ef103e8a" (UID: "525c4513-e445-4135-9766-9bd1ef103e8a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:32:51.267316 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.267291    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "525c4513-e445-4135-9766-9bd1ef103e8a" (UID: "525c4513-e445-4135-9766-9bd1ef103e8a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:51.267316 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.267295    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525c4513-e445-4135-9766-9bd1ef103e8a-kube-api-access-lp5dl" (OuterVolumeSpecName: "kube-api-access-lp5dl") pod "525c4513-e445-4135-9766-9bd1ef103e8a" (UID: "525c4513-e445-4135-9766-9bd1ef103e8a"). InnerVolumeSpecName "kube-api-access-lp5dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:32:51.267486 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.267468    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "525c4513-e445-4135-9766-9bd1ef103e8a" (UID: "525c4513-e445-4135-9766-9bd1ef103e8a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:32:51.366347 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.366304    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:51.366347 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.366343    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:51.366347 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.366353    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp5dl\" (UniqueName: \"kubernetes.io/projected/525c4513-e445-4135-9766-9bd1ef103e8a-kube-api-access-lp5dl\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:51.366347 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.366363    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:51.366592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.366371    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:51.366592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.366379    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/525c4513-e445-4135-9766-9bd1ef103e8a-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:51.366592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.366387    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/525c4513-e445-4135-9766-9bd1ef103e8a-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:32:51.915842 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.915809    2586 generic.go:356] "Generic (PLEG): container finished" podID="525c4513-e445-4135-9766-9bd1ef103e8a" containerID="65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5" exitCode=0
Sep 09 10:32:51.916255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.915851    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cfb7697-sjhff"
Sep 09 10:32:51.916255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.915905    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cfb7697-sjhff" event={"ID":"525c4513-e445-4135-9766-9bd1ef103e8a","Type":"ContainerDied","Data":"65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5"}
Sep 09 10:32:51.916255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.915960    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cfb7697-sjhff" event={"ID":"525c4513-e445-4135-9766-9bd1ef103e8a","Type":"ContainerDied","Data":"6a99e2d0ad93a9a107c842b54f17f9e339f274eb0c25d6f0be99f463d371c1c4"}
Sep 09 10:32:51.916255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.915987    2586 scope.go:117] "RemoveContainer" containerID="65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5"
Sep 09 10:32:51.924613 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.924596    2586 scope.go:117] "RemoveContainer" containerID="65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5"
Sep 09 10:32:51.924875 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:32:51.924855    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5\": container with ID starting with 65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5 not found: ID does not exist" containerID="65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5"
Sep 09 10:32:51.924942 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.924884    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5"} err="failed to get container status \"65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5\": rpc error: code = NotFound desc = could not find container \"65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5\": container with ID starting with 65807c8252e2396dc97b5cf4ce9444afb620fbbbeb072b250b3f499ac31566d5 not found: ID does not exist"
Sep 09 10:32:51.937436 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.937411    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66cfb7697-sjhff"]
Sep 09 10:32:51.942261 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:51.942241    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66cfb7697-sjhff"]
Sep 09 10:32:52.473796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:32:52.473760    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="525c4513-e445-4135-9766-9bd1ef103e8a" path="/var/lib/kubelet/pods/525c4513-e445-4135-9766-9bd1ef103e8a/volumes"
Sep 09 10:33:28.126349 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.126266    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dff8d754b-k76vm"]
Sep 09 10:33:28.126798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.126669    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525c4513-e445-4135-9766-9bd1ef103e8a" containerName="console"
Sep 09 10:33:28.126798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.126682    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="525c4513-e445-4135-9766-9bd1ef103e8a" containerName="console"
Sep 09 10:33:28.126798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.126741    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="525c4513-e445-4135-9766-9bd1ef103e8a" containerName="console"
Sep 09 10:33:28.129689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.129673    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.140217 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.140184    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dff8d754b-k76vm"]
Sep 09 10:33:28.191152 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.191120    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-oauth-serving-cert\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.191326 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.191165    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-trusted-ca-bundle\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.191326 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.191219    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-oauth-config\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.191326 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.191246    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-console-config\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.191326 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.191296    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-service-ca\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.191483 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.191368    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rq78\" (UniqueName: \"kubernetes.io/projected/78c6242d-b3eb-4131-9816-917e2951b31e-kube-api-access-4rq78\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.191483 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.191403    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-serving-cert\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.292184 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.292157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-oauth-config\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.292359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.292192    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-console-config\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.292359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.292207    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-service-ca\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.292359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.292266    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rq78\" (UniqueName: \"kubernetes.io/projected/78c6242d-b3eb-4131-9816-917e2951b31e-kube-api-access-4rq78\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.292516 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.292436    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-serving-cert\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.292576 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.292531    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-oauth-serving-cert\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.292627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.292590    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-trusted-ca-bundle\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.293136 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.293108    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-console-config\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.293271 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.293190    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-service-ca\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.293342 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.293287    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-oauth-serving-cert\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.293854 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.293825    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-trusted-ca-bundle\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.294808 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.294787    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-oauth-config\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.294988 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.294972    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-serving-cert\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.300680 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.300651    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rq78\" (UniqueName: \"kubernetes.io/projected/78c6242d-b3eb-4131-9816-917e2951b31e-kube-api-access-4rq78\") pod \"console-5dff8d754b-k76vm\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") " pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.439347 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.439272    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:28.716050 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:28.716025    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dff8d754b-k76vm"]
Sep 09 10:33:28.717880 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:33:28.717853    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c6242d_b3eb_4131_9816_917e2951b31e.slice/crio-33d80356555592c9113e2000e558ec6a200401740e89f81b12ac375eb5ce9466 WatchSource:0}: Error finding container 33d80356555592c9113e2000e558ec6a200401740e89f81b12ac375eb5ce9466: Status 404 returned error can't find the container with id 33d80356555592c9113e2000e558ec6a200401740e89f81b12ac375eb5ce9466
Sep 09 10:33:29.069727 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:29.069634    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dff8d754b-k76vm" event={"ID":"78c6242d-b3eb-4131-9816-917e2951b31e","Type":"ContainerStarted","Data":"e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1"}
Sep 09 10:33:29.069727 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:29.069692    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dff8d754b-k76vm" event={"ID":"78c6242d-b3eb-4131-9816-917e2951b31e","Type":"ContainerStarted","Data":"33d80356555592c9113e2000e558ec6a200401740e89f81b12ac375eb5ce9466"}
Sep 09 10:33:29.090848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:29.090779    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dff8d754b-k76vm" podStartSLOduration=1.090762885 podStartE2EDuration="1.090762885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:33:29.089093648 +0000 UTC m=+451.144652707" watchObservedRunningTime="2026-09-09 10:33:29.090762885 +0000 UTC m=+451.146321934"
Sep 09 10:33:38.440360 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:38.440315    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:38.440739 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:38.440384    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:38.445105 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:38.445085    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:39.116844 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:39.116814    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:33:39.168981 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:33:39.168945    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b7dcfd77-twktd"]
Sep 09 10:34:04.188680 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.188642    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b7dcfd77-twktd" podUID="b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" containerName="console" containerID="cri-o://6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a" gracePeriod=15
Sep 09 10:34:04.428884 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.428860    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:34:04.488224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488154    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-config\") pod \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") "
Sep 09 10:34:04.488224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488197    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrg9k\" (UniqueName: \"kubernetes.io/projected/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-kube-api-access-zrg9k\") pod \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") "
Sep 09 10:34:04.488224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488216    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-service-ca\") pod \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") "
Sep 09 10:34:04.488477 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488251    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-oauth-config\") pod \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") "
Sep 09 10:34:04.488477 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488279    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-trusted-ca-bundle\") pod \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") "
Sep 09 10:34:04.488477 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488310    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-oauth-serving-cert\") pod \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") "
Sep 09 10:34:04.488477 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488357    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-serving-cert\") pod \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\" (UID: \"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e\") "
Sep 09 10:34:04.488714 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488538    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-config" (OuterVolumeSpecName: "console-config") pod "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" (UID: "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:34:04.488770 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488745    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-service-ca" (OuterVolumeSpecName: "service-ca") pod "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" (UID: "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:34:04.488820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488762    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" (UID: "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:34:04.488820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.488751    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" (UID: "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:34:04.490372 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.490346    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-kube-api-access-zrg9k" (OuterVolumeSpecName: "kube-api-access-zrg9k") pod "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" (UID: "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e"). InnerVolumeSpecName "kube-api-access-zrg9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:34:04.490530 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.490507    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" (UID: "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:34:04.490576 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.490521    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" (UID: "b86b746e-dbc5-4e50-8a46-6ca5030b6d2e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:34:04.589105 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.589065    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:34:04.589105 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.589099    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrg9k\" (UniqueName: \"kubernetes.io/projected/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-kube-api-access-zrg9k\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:34:04.589105 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.589111    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:34:04.589335 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.589120    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:34:04.589335 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.589129    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:34:04.589335 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.589137    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:34:04.589335 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:04.589145    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:34:05.215071 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.215033    2586 generic.go:356] "Generic (PLEG): container finished" podID="b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" containerID="6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a" exitCode=0
Sep 09 10:34:05.215071 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.215074    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7dcfd77-twktd" event={"ID":"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e","Type":"ContainerDied","Data":"6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a"}
Sep 09 10:34:05.215526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.215111    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7dcfd77-twktd" event={"ID":"b86b746e-dbc5-4e50-8a46-6ca5030b6d2e","Type":"ContainerDied","Data":"96d8a0d06f4cd45fcaf2b6154ac1de9685e2a446fd3c5dbcff8f0b60d153f390"}
Sep 09 10:34:05.215526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.215132    2586 scope.go:117] "RemoveContainer" containerID="6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a"
Sep 09 10:34:05.215526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.215076    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7dcfd77-twktd"
Sep 09 10:34:05.223694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.223678    2586 scope.go:117] "RemoveContainer" containerID="6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a"
Sep 09 10:34:05.223987 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:34:05.223968    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a\": container with ID starting with 6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a not found: ID does not exist" containerID="6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a"
Sep 09 10:34:05.224061 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.223993    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a"} err="failed to get container status \"6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a\": rpc error: code = NotFound desc = could not find container \"6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a\": container with ID starting with 6f6fdc5bba90c58519aacc138ee4b87063bbdcbf3b662df0bf066b108521f80a not found: ID does not exist"
Sep 09 10:34:05.237770 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.237746    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b7dcfd77-twktd"]
Sep 09 10:34:05.242688 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:05.242666    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b7dcfd77-twktd"]
Sep 09 10:34:06.474473 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:34:06.474437    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" path="/var/lib/kubelet/pods/b86b746e-dbc5-4e50-8a46-6ca5030b6d2e/volumes"
Sep 09 10:35:47.194077 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.194040    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"]
Sep 09 10:35:47.194659 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.194366    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" containerName="console"
Sep 09 10:35:47.194659 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.194377    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" containerName="console"
Sep 09 10:35:47.194659 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.194445    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86b746e-dbc5-4e50-8a46-6ca5030b6d2e" containerName="console"
Sep 09 10:35:47.197740 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.197723    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.200142 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.200122    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 10:35:47.200319 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.200300    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bccwt\""
Sep 09 10:35:47.200820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.200806    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 10:35:47.216516 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.216491    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"]
Sep 09 10:35:47.270095 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.270063    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.270095 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.270095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6spl\" (UniqueName: \"kubernetes.io/projected/6091ed8f-c547-488b-9c1c-25d22bb15f76-kube-api-access-p6spl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.270276 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.270136    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.370572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.370533    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.370572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.370576    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6spl\" (UniqueName: \"kubernetes.io/projected/6091ed8f-c547-488b-9c1c-25d22bb15f76-kube-api-access-p6spl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.370727 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.370609    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.370958 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.370938    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.371028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.371008    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.379243 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.379218    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6spl\" (UniqueName: \"kubernetes.io/projected/6091ed8f-c547-488b-9c1c-25d22bb15f76-kube-api-access-p6spl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.506858 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.506777    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:35:47.776067 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:47.775935    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"]
Sep 09 10:35:47.778227 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:35:47.778200    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6091ed8f_c547_488b_9c1c_25d22bb15f76.slice/crio-5b017b18fbd9c47c9437e28ce108ae10a2ec21d75e27d3ba236c65eaa50cf328 WatchSource:0}: Error finding container 5b017b18fbd9c47c9437e28ce108ae10a2ec21d75e27d3ba236c65eaa50cf328: Status 404 returned error can't find the container with id 5b017b18fbd9c47c9437e28ce108ae10a2ec21d75e27d3ba236c65eaa50cf328
Sep 09 10:35:48.628535 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:48.628496    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x" event={"ID":"6091ed8f-c547-488b-9c1c-25d22bb15f76","Type":"ContainerStarted","Data":"5b017b18fbd9c47c9437e28ce108ae10a2ec21d75e27d3ba236c65eaa50cf328"}
Sep 09 10:35:53.945848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:53.945818    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7755c8cfcb-7ghqf"]
Sep 09 10:35:53.949047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:53.949032    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:53.961426 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:53.961404    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7755c8cfcb-7ghqf"]
Sep 09 10:35:54.031862 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.031791    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xpf\" (UniqueName: \"kubernetes.io/projected/5721e109-5915-4438-b897-4ac7a9c5458e-kube-api-access-g9xpf\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.031862 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.031834    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-trusted-ca-bundle\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.032048 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.031869    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-console-config\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.032048 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.031903    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-oauth-serving-cert\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.032048 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.031920    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5721e109-5915-4438-b897-4ac7a9c5458e-console-serving-cert\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.032048 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.032017    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5721e109-5915-4438-b897-4ac7a9c5458e-console-oauth-config\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.032171 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.032086    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-service-ca\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133017 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.132986    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-trusted-ca-bundle\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133026    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-console-config\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133048    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-oauth-serving-cert\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133068    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5721e109-5915-4438-b897-4ac7a9c5458e-console-serving-cert\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133092    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5721e109-5915-4438-b897-4ac7a9c5458e-console-oauth-config\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133148    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-service-ca\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133443 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133290    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xpf\" (UniqueName: \"kubernetes.io/projected/5721e109-5915-4438-b897-4ac7a9c5458e-kube-api-access-g9xpf\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133817 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133787    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-oauth-serving-cert\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.133960 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133789    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-console-config\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.134028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.133996    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-trusted-ca-bundle\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.134181 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.134162    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5721e109-5915-4438-b897-4ac7a9c5458e-service-ca\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.135756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.135728    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5721e109-5915-4438-b897-4ac7a9c5458e-console-oauth-config\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.135756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.135750    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5721e109-5915-4438-b897-4ac7a9c5458e-console-serving-cert\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.142008 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.141986    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xpf\" (UniqueName: \"kubernetes.io/projected/5721e109-5915-4438-b897-4ac7a9c5458e-kube-api-access-g9xpf\") pod \"console-7755c8cfcb-7ghqf\" (UID: \"5721e109-5915-4438-b897-4ac7a9c5458e\") " pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.257443 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.257409    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:35:54.529638 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.529602    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7755c8cfcb-7ghqf"]
Sep 09 10:35:54.533380 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:35:54.533349    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5721e109_5915_4438_b897_4ac7a9c5458e.slice/crio-2724bde8a60e9fbb67e32e972dd676cace5d8322b8a7e52d1a985eef8e5bc940 WatchSource:0}: Error finding container 2724bde8a60e9fbb67e32e972dd676cace5d8322b8a7e52d1a985eef8e5bc940: Status 404 returned error can't find the container with id 2724bde8a60e9fbb67e32e972dd676cace5d8322b8a7e52d1a985eef8e5bc940
Sep 09 10:35:54.656599 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.656559    2586 generic.go:356] "Generic (PLEG): container finished" podID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerID="d8cd4f8ec812868101210280509b9128f502a8186b5f8fb6cbb332528a9b8cf6" exitCode=0
Sep 09 10:35:54.656757 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.656616    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x" event={"ID":"6091ed8f-c547-488b-9c1c-25d22bb15f76","Type":"ContainerDied","Data":"d8cd4f8ec812868101210280509b9128f502a8186b5f8fb6cbb332528a9b8cf6"}
Sep 09 10:35:54.658638 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.658612    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7755c8cfcb-7ghqf" event={"ID":"5721e109-5915-4438-b897-4ac7a9c5458e","Type":"ContainerStarted","Data":"c22801b6d1083c7c6cdaa7e22668324cacf72c43e244f96142990478521fc083"}
Sep 09 10:35:54.658732 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.658655    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7755c8cfcb-7ghqf" event={"ID":"5721e109-5915-4438-b897-4ac7a9c5458e","Type":"ContainerStarted","Data":"2724bde8a60e9fbb67e32e972dd676cace5d8322b8a7e52d1a985eef8e5bc940"}
Sep 09 10:35:54.692782 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:54.692742    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7755c8cfcb-7ghqf" podStartSLOduration=1.69272949 podStartE2EDuration="1.69272949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:35:54.692530594 +0000 UTC m=+596.748089644" watchObservedRunningTime="2026-09-09 10:35:54.69272949 +0000 UTC m=+596.748288540"
Sep 09 10:35:56.672108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:56.672080    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x" event={"ID":"6091ed8f-c547-488b-9c1c-25d22bb15f76","Type":"ContainerStarted","Data":"3035b9aa6971675887e1ba6b7959ced23e0219fa2199f086a952fa383f6efb37"}
Sep 09 10:35:57.678056 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:57.678020    2586 generic.go:356] "Generic (PLEG): container finished" podID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerID="3035b9aa6971675887e1ba6b7959ced23e0219fa2199f086a952fa383f6efb37" exitCode=0
Sep 09 10:35:57.678385 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:35:57.678108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x" event={"ID":"6091ed8f-c547-488b-9c1c-25d22bb15f76","Type":"ContainerDied","Data":"3035b9aa6971675887e1ba6b7959ced23e0219fa2199f086a952fa383f6efb37"}
Sep 09 10:36:04.258556 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:04.258524    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:36:04.258875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:04.258635    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:36:04.263721 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:04.263700    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:36:04.711074 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:04.711037    2586 generic.go:356] "Generic (PLEG): container finished" podID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerID="7fdb6bc5fbb7b87d58791d6ebf25d8cca53a77a523ad6f2c4a6aede5cc11da0d" exitCode=0
Sep 09 10:36:04.711272 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:04.711121    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x" event={"ID":"6091ed8f-c547-488b-9c1c-25d22bb15f76","Type":"ContainerDied","Data":"7fdb6bc5fbb7b87d58791d6ebf25d8cca53a77a523ad6f2c4a6aede5cc11da0d"}
Sep 09 10:36:04.716143 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:04.716121    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7755c8cfcb-7ghqf"
Sep 09 10:36:04.792983 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:04.792950    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dff8d754b-k76vm"]
Sep 09 10:36:05.836988 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:05.836965    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:36:05.936145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:05.936109    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6spl\" (UniqueName: \"kubernetes.io/projected/6091ed8f-c547-488b-9c1c-25d22bb15f76-kube-api-access-p6spl\") pod \"6091ed8f-c547-488b-9c1c-25d22bb15f76\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") "
Sep 09 10:36:05.936340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:05.936199    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-bundle\") pod \"6091ed8f-c547-488b-9c1c-25d22bb15f76\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") "
Sep 09 10:36:05.936340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:05.936289    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-util\") pod \"6091ed8f-c547-488b-9c1c-25d22bb15f76\" (UID: \"6091ed8f-c547-488b-9c1c-25d22bb15f76\") "
Sep 09 10:36:05.936879 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:05.936855    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-bundle" (OuterVolumeSpecName: "bundle") pod "6091ed8f-c547-488b-9c1c-25d22bb15f76" (UID: "6091ed8f-c547-488b-9c1c-25d22bb15f76"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:36:05.938388 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:05.938349    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6091ed8f-c547-488b-9c1c-25d22bb15f76-kube-api-access-p6spl" (OuterVolumeSpecName: "kube-api-access-p6spl") pod "6091ed8f-c547-488b-9c1c-25d22bb15f76" (UID: "6091ed8f-c547-488b-9c1c-25d22bb15f76"). InnerVolumeSpecName "kube-api-access-p6spl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:36:05.940503 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:05.940485    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-util" (OuterVolumeSpecName: "util") pod "6091ed8f-c547-488b-9c1c-25d22bb15f76" (UID: "6091ed8f-c547-488b-9c1c-25d22bb15f76"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:36:06.037872 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:06.037779    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:06.037872 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:06.037819    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6091ed8f-c547-488b-9c1c-25d22bb15f76-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:06.037872 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:06.037829    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6spl\" (UniqueName: \"kubernetes.io/projected/6091ed8f-c547-488b-9c1c-25d22bb15f76-kube-api-access-p6spl\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:06.721381 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:06.721351    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x"
Sep 09 10:36:06.721381 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:06.721372    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mr5x" event={"ID":"6091ed8f-c547-488b-9c1c-25d22bb15f76","Type":"ContainerDied","Data":"5b017b18fbd9c47c9437e28ce108ae10a2ec21d75e27d3ba236c65eaa50cf328"}
Sep 09 10:36:06.721577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:06.721414    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b017b18fbd9c47c9437e28ce108ae10a2ec21d75e27d3ba236c65eaa50cf328"
Sep 09 10:36:10.363954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.363917    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"]
Sep 09 10:36:10.364306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.364252    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerName="pull"
Sep 09 10:36:10.364306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.364263    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerName="pull"
Sep 09 10:36:10.364306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.364282    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerName="util"
Sep 09 10:36:10.364306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.364288    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerName="util"
Sep 09 10:36:10.364306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.364296    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerName="extract"
Sep 09 10:36:10.364306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.364302    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerName="extract"
Sep 09 10:36:10.364502 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.364364    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6091ed8f-c547-488b-9c1c-25d22bb15f76" containerName="extract"
Sep 09 10:36:10.368002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.367986    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"
Sep 09 10:36:10.370828 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.370804    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 10:36:10.370981 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.370838    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:36:10.370981 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.370869    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lfj2k\""
Sep 09 10:36:10.377768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.377749    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6bd444-de2d-4b4f-9669-955aca52abca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-k9mvz\" (UID: \"6e6bd444-de2d-4b4f-9669-955aca52abca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"
Sep 09 10:36:10.377866 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.377776    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gvx\" (UniqueName: \"kubernetes.io/projected/6e6bd444-de2d-4b4f-9669-955aca52abca-kube-api-access-m9gvx\") pod \"cert-manager-operator-controller-manager-55c444bf86-k9mvz\" (UID: \"6e6bd444-de2d-4b4f-9669-955aca52abca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"
Sep 09 10:36:10.383857 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.383836    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"]
Sep 09 10:36:10.479213 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.479180    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6bd444-de2d-4b4f-9669-955aca52abca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-k9mvz\" (UID: \"6e6bd444-de2d-4b4f-9669-955aca52abca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"
Sep 09 10:36:10.479213 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.479213    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gvx\" (UniqueName: \"kubernetes.io/projected/6e6bd444-de2d-4b4f-9669-955aca52abca-kube-api-access-m9gvx\") pod \"cert-manager-operator-controller-manager-55c444bf86-k9mvz\" (UID: \"6e6bd444-de2d-4b4f-9669-955aca52abca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"
Sep 09 10:36:10.479529 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.479511    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6bd444-de2d-4b4f-9669-955aca52abca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-k9mvz\" (UID: \"6e6bd444-de2d-4b4f-9669-955aca52abca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"
Sep 09 10:36:10.488020 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.487996    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gvx\" (UniqueName: \"kubernetes.io/projected/6e6bd444-de2d-4b4f-9669-955aca52abca-kube-api-access-m9gvx\") pod \"cert-manager-operator-controller-manager-55c444bf86-k9mvz\" (UID: \"6e6bd444-de2d-4b4f-9669-955aca52abca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"
Sep 09 10:36:10.676927 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.676828    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"
Sep 09 10:36:10.952377 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:10.952352    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz"]
Sep 09 10:36:10.954558 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:36:10.954531    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6bd444_de2d_4b4f_9669_955aca52abca.slice/crio-8b3a1b19e450aa4dcd2b0c4223c8ab2e61a0a1f8ecdcb46cf38dde09aa731081 WatchSource:0}: Error finding container 8b3a1b19e450aa4dcd2b0c4223c8ab2e61a0a1f8ecdcb46cf38dde09aa731081: Status 404 returned error can't find the container with id 8b3a1b19e450aa4dcd2b0c4223c8ab2e61a0a1f8ecdcb46cf38dde09aa731081
Sep 09 10:36:11.743223 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:11.743191    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz" event={"ID":"6e6bd444-de2d-4b4f-9669-955aca52abca","Type":"ContainerStarted","Data":"8b3a1b19e450aa4dcd2b0c4223c8ab2e61a0a1f8ecdcb46cf38dde09aa731081"}
Sep 09 10:36:13.754143 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:13.754108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz" event={"ID":"6e6bd444-de2d-4b4f-9669-955aca52abca","Type":"ContainerStarted","Data":"fbe682638206480c2fa280c018aa54bca10a75dfd37e97559fc9b3396d8061c4"}
Sep 09 10:36:13.776065 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:13.776019    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k9mvz" podStartSLOduration=1.8698891579999999 podStartE2EDuration="3.776003472s" totalImagesPullingTime="1.906114314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:36:13.774761491 +0000 UTC m=+615.830320542" watchObservedRunningTime="2026-09-09 10:36:13.776003472 +0000 UTC m=+615.831562522"
Sep 09 10:36:14.963167 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:14.963133    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"]
Sep 09 10:36:14.966899 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:14.966871    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:14.969693 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:14.969670    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 10:36:14.969876 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:14.969840    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 10:36:14.970586 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:14.970571    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bccwt\""
Sep 09 10:36:14.976475 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:14.976451    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"]
Sep 09 10:36:15.013235 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.013208    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.013423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.013265    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29tlj\" (UniqueName: \"kubernetes.io/projected/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-kube-api-access-29tlj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.013423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.013380    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.114700 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.114670    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.114874 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.114707    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.114874 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.114749    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29tlj\" (UniqueName: \"kubernetes.io/projected/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-kube-api-access-29tlj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.115074 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.115052    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.115140 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.115097    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.124384 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.124363    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29tlj\" (UniqueName: \"kubernetes.io/projected/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-kube-api-access-29tlj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.276260 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.276183    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:15.549360 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.549279    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"]
Sep 09 10:36:15.552360 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:36:15.552319    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2d7aa6_e315_4d55_bf6e_f30f4c396d94.slice/crio-b14adac8589794f9b81538dd386be31e6f2ffebfb1e204475220550c61f71fff WatchSource:0}: Error finding container b14adac8589794f9b81538dd386be31e6f2ffebfb1e204475220550c61f71fff: Status 404 returned error can't find the container with id b14adac8589794f9b81538dd386be31e6f2ffebfb1e204475220550c61f71fff
Sep 09 10:36:15.764537 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.764505    2586 generic.go:356] "Generic (PLEG): container finished" podID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerID="f6324a885af76624ea1885265092f1c4c154355ce236eb0d7a019e7e3483b858" exitCode=0
Sep 09 10:36:15.764692 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.764599    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp" event={"ID":"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94","Type":"ContainerDied","Data":"f6324a885af76624ea1885265092f1c4c154355ce236eb0d7a019e7e3483b858"}
Sep 09 10:36:15.764692 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:15.764655    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp" event={"ID":"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94","Type":"ContainerStarted","Data":"b14adac8589794f9b81538dd386be31e6f2ffebfb1e204475220550c61f71fff"}
Sep 09 10:36:16.339591 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.339561    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"]
Sep 09 10:36:16.342748 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.342732    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"
Sep 09 10:36:16.346928 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.346878    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-f4mch\""
Sep 09 10:36:16.347070 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.347054    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 10:36:16.347155 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.347141    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 10:36:16.355115 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.355094    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"]
Sep 09 10:36:16.427175 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.427097    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56z2c\" (UniqueName: \"kubernetes.io/projected/674e9a9b-7277-4c2d-a03c-d4ac10f4572d-kube-api-access-56z2c\") pod \"cert-manager-cainjector-559cdd8c87-jbqxx\" (UID: \"674e9a9b-7277-4c2d-a03c-d4ac10f4572d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"
Sep 09 10:36:16.427175 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.427155    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/674e9a9b-7277-4c2d-a03c-d4ac10f4572d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jbqxx\" (UID: \"674e9a9b-7277-4c2d-a03c-d4ac10f4572d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"
Sep 09 10:36:16.528303 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.528266    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56z2c\" (UniqueName: \"kubernetes.io/projected/674e9a9b-7277-4c2d-a03c-d4ac10f4572d-kube-api-access-56z2c\") pod \"cert-manager-cainjector-559cdd8c87-jbqxx\" (UID: \"674e9a9b-7277-4c2d-a03c-d4ac10f4572d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"
Sep 09 10:36:16.528625 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.528597    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/674e9a9b-7277-4c2d-a03c-d4ac10f4572d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jbqxx\" (UID: \"674e9a9b-7277-4c2d-a03c-d4ac10f4572d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"
Sep 09 10:36:16.536707 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.536682    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/674e9a9b-7277-4c2d-a03c-d4ac10f4572d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jbqxx\" (UID: \"674e9a9b-7277-4c2d-a03c-d4ac10f4572d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"
Sep 09 10:36:16.536830 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.536767    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56z2c\" (UniqueName: \"kubernetes.io/projected/674e9a9b-7277-4c2d-a03c-d4ac10f4572d-kube-api-access-56z2c\") pod \"cert-manager-cainjector-559cdd8c87-jbqxx\" (UID: \"674e9a9b-7277-4c2d-a03c-d4ac10f4572d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"
Sep 09 10:36:16.665795 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.665758    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"
Sep 09 10:36:16.946510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:16.946443    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx"]
Sep 09 10:36:16.949073 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:36:16.949049    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod674e9a9b_7277_4c2d_a03c_d4ac10f4572d.slice/crio-ce5102955627dda2ed6ee655009f1145031275c2eadc985c5c755c8b22765321 WatchSource:0}: Error finding container ce5102955627dda2ed6ee655009f1145031275c2eadc985c5c755c8b22765321: Status 404 returned error can't find the container with id ce5102955627dda2ed6ee655009f1145031275c2eadc985c5c755c8b22765321
Sep 09 10:36:17.779099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:17.779063    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx" event={"ID":"674e9a9b-7277-4c2d-a03c-d4ac10f4572d","Type":"ContainerStarted","Data":"ce5102955627dda2ed6ee655009f1145031275c2eadc985c5c755c8b22765321"}
Sep 09 10:36:18.786166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:18.786132    2586 generic.go:356] "Generic (PLEG): container finished" podID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerID="93ea4bc72b8bbe84d422dac71922d6effe461dbe2a72028f0a32971bb5dbdefb" exitCode=0
Sep 09 10:36:18.786593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:18.786192    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp" event={"ID":"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94","Type":"ContainerDied","Data":"93ea4bc72b8bbe84d422dac71922d6effe461dbe2a72028f0a32971bb5dbdefb"}
Sep 09 10:36:19.798529 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:19.798496    2586 generic.go:356] "Generic (PLEG): container finished" podID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerID="bea02b5b3ead9a378c295c5ffa9309bbcd12b81cb7ed476460e10865eb6b222e" exitCode=0
Sep 09 10:36:19.798965 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:19.798533    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp" event={"ID":"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94","Type":"ContainerDied","Data":"bea02b5b3ead9a378c295c5ffa9309bbcd12b81cb7ed476460e10865eb6b222e"}
Sep 09 10:36:20.805635 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.805546    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx" event={"ID":"674e9a9b-7277-4c2d-a03c-d4ac10f4572d","Type":"ContainerStarted","Data":"4691ecffaa1a976877ef9b78753e5791c1410cf321c95be9f5e3bc6a307b42ac"}
Sep 09 10:36:20.822829 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.822771    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jbqxx" podStartSLOduration=1.223427829 podStartE2EDuration="4.822750868s" totalImagesPullingTime="3.599323039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:36:20.821976432 +0000 UTC m=+622.877535508" watchObservedRunningTime="2026-09-09 10:36:20.822750868 +0000 UTC m=+622.878309937"
Sep 09 10:36:20.937098 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.937072    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:20.970524 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.970496    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-bundle\") pod \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") "
Sep 09 10:36:20.970659 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.970552    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-util\") pod \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") "
Sep 09 10:36:20.970659 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.970595    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29tlj\" (UniqueName: \"kubernetes.io/projected/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-kube-api-access-29tlj\") pod \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\" (UID: \"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94\") "
Sep 09 10:36:20.970922 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.970873    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-bundle" (OuterVolumeSpecName: "bundle") pod "0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" (UID: "0a2d7aa6-e315-4d55-bf6e-f30f4c396d94"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:36:20.972907 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.972855    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-kube-api-access-29tlj" (OuterVolumeSpecName: "kube-api-access-29tlj") pod "0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" (UID: "0a2d7aa6-e315-4d55-bf6e-f30f4c396d94"). InnerVolumeSpecName "kube-api-access-29tlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:36:20.978427 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:20.978400    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-util" (OuterVolumeSpecName: "util") pod "0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" (UID: "0a2d7aa6-e315-4d55-bf6e-f30f4c396d94"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:36:21.071662 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:21.071583    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:21.071662 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:21.071610    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29tlj\" (UniqueName: \"kubernetes.io/projected/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-kube-api-access-29tlj\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:21.071662 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:21.071620    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2d7aa6-e315-4d55-bf6e-f30f4c396d94-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:21.812469 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:21.812429    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp" event={"ID":"0a2d7aa6-e315-4d55-bf6e-f30f4c396d94","Type":"ContainerDied","Data":"b14adac8589794f9b81538dd386be31e6f2ffebfb1e204475220550c61f71fff"}
Sep 09 10:36:21.812469 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:21.812458    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv6twp"
Sep 09 10:36:21.813022 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:21.812476    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14adac8589794f9b81538dd386be31e6f2ffebfb1e204475220550c61f71fff"
Sep 09 10:36:28.119196 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.119165    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"]
Sep 09 10:36:28.119627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.119489    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerName="pull"
Sep 09 10:36:28.119627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.119500    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerName="pull"
Sep 09 10:36:28.119627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.119525    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerName="extract"
Sep 09 10:36:28.119627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.119531    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerName="extract"
Sep 09 10:36:28.119627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.119539    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerName="util"
Sep 09 10:36:28.119627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.119544    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerName="util"
Sep 09 10:36:28.119627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.119592    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2d7aa6-e315-4d55-bf6e-f30f4c396d94" containerName="extract"
Sep 09 10:36:28.122304 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.122288    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"
Sep 09 10:36:28.124403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.124380    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-sdv9m\""
Sep 09 10:36:28.124403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.124396    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 10:36:28.124873 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.124855    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 10:36:28.131005 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.130981    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"]
Sep 09 10:36:28.236827 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.236795    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9bsb\" (UniqueName: \"kubernetes.io/projected/72d8059e-141a-46a3-ac60-7f7b45fcf36d-kube-api-access-n9bsb\") pod \"openshift-lws-operator-fd8ccff4c-gxvwn\" (UID: \"72d8059e-141a-46a3-ac60-7f7b45fcf36d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"
Sep 09 10:36:28.236993 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.236856    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72d8059e-141a-46a3-ac60-7f7b45fcf36d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gxvwn\" (UID: \"72d8059e-141a-46a3-ac60-7f7b45fcf36d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"
Sep 09 10:36:28.337934 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.337886    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9bsb\" (UniqueName: \"kubernetes.io/projected/72d8059e-141a-46a3-ac60-7f7b45fcf36d-kube-api-access-n9bsb\") pod \"openshift-lws-operator-fd8ccff4c-gxvwn\" (UID: \"72d8059e-141a-46a3-ac60-7f7b45fcf36d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"
Sep 09 10:36:28.338069 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.337975    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72d8059e-141a-46a3-ac60-7f7b45fcf36d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gxvwn\" (UID: \"72d8059e-141a-46a3-ac60-7f7b45fcf36d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"
Sep 09 10:36:28.338301 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.338284    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72d8059e-141a-46a3-ac60-7f7b45fcf36d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gxvwn\" (UID: \"72d8059e-141a-46a3-ac60-7f7b45fcf36d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"
Sep 09 10:36:28.347713 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.347689    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9bsb\" (UniqueName: \"kubernetes.io/projected/72d8059e-141a-46a3-ac60-7f7b45fcf36d-kube-api-access-n9bsb\") pod \"openshift-lws-operator-fd8ccff4c-gxvwn\" (UID: \"72d8059e-141a-46a3-ac60-7f7b45fcf36d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"
Sep 09 10:36:28.432280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.432229    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"
Sep 09 10:36:28.711562 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.711540    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn"]
Sep 09 10:36:28.713488 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:36:28.713461    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d8059e_141a_46a3_ac60_7f7b45fcf36d.slice/crio-50c91ee4f765023d9a11da16e17f24e5a9ede038a3e06a07f534ef0fe4f7b3d4 WatchSource:0}: Error finding container 50c91ee4f765023d9a11da16e17f24e5a9ede038a3e06a07f534ef0fe4f7b3d4: Status 404 returned error can't find the container with id 50c91ee4f765023d9a11da16e17f24e5a9ede038a3e06a07f534ef0fe4f7b3d4
Sep 09 10:36:28.852312 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:28.852278    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn" event={"ID":"72d8059e-141a-46a3-ac60-7f7b45fcf36d","Type":"ContainerStarted","Data":"50c91ee4f765023d9a11da16e17f24e5a9ede038a3e06a07f534ef0fe4f7b3d4"}
Sep 09 10:36:29.812693 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:29.812634    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dff8d754b-k76vm" podUID="78c6242d-b3eb-4131-9816-917e2951b31e" containerName="console" containerID="cri-o://e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1" gracePeriod=15
Sep 09 10:36:30.054320 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.054292    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:36:30.153115 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153083    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-oauth-serving-cert\") pod \"78c6242d-b3eb-4131-9816-917e2951b31e\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") "
Sep 09 10:36:30.153279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153146    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-oauth-config\") pod \"78c6242d-b3eb-4131-9816-917e2951b31e\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") "
Sep 09 10:36:30.153279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153174    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-trusted-ca-bundle\") pod \"78c6242d-b3eb-4131-9816-917e2951b31e\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") "
Sep 09 10:36:30.153279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153195    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-serving-cert\") pod \"78c6242d-b3eb-4131-9816-917e2951b31e\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") "
Sep 09 10:36:30.153279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153240    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-service-ca\") pod \"78c6242d-b3eb-4131-9816-917e2951b31e\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") "
Sep 09 10:36:30.153480 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153370    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rq78\" (UniqueName: \"kubernetes.io/projected/78c6242d-b3eb-4131-9816-917e2951b31e-kube-api-access-4rq78\") pod \"78c6242d-b3eb-4131-9816-917e2951b31e\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") "
Sep 09 10:36:30.153480 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153463    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-console-config\") pod \"78c6242d-b3eb-4131-9816-917e2951b31e\" (UID: \"78c6242d-b3eb-4131-9816-917e2951b31e\") "
Sep 09 10:36:30.153614 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153587    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78c6242d-b3eb-4131-9816-917e2951b31e" (UID: "78c6242d-b3eb-4131-9816-917e2951b31e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:36:30.153672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153644    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-service-ca" (OuterVolumeSpecName: "service-ca") pod "78c6242d-b3eb-4131-9816-917e2951b31e" (UID: "78c6242d-b3eb-4131-9816-917e2951b31e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:36:30.153795 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153770    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "78c6242d-b3eb-4131-9816-917e2951b31e" (UID: "78c6242d-b3eb-4131-9816-917e2951b31e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:36:30.153876 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153861    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:30.153964 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153882    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:30.153964 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153917    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-console-config" (OuterVolumeSpecName: "console-config") pod "78c6242d-b3eb-4131-9816-917e2951b31e" (UID: "78c6242d-b3eb-4131-9816-917e2951b31e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:36:30.153964 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.153929    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:30.155452 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.155431    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c6242d-b3eb-4131-9816-917e2951b31e-kube-api-access-4rq78" (OuterVolumeSpecName: "kube-api-access-4rq78") pod "78c6242d-b3eb-4131-9816-917e2951b31e" (UID: "78c6242d-b3eb-4131-9816-917e2951b31e"). InnerVolumeSpecName "kube-api-access-4rq78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:36:30.155514 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.155458    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78c6242d-b3eb-4131-9816-917e2951b31e" (UID: "78c6242d-b3eb-4131-9816-917e2951b31e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:36:30.155552 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.155534    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78c6242d-b3eb-4131-9816-917e2951b31e" (UID: "78c6242d-b3eb-4131-9816-917e2951b31e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:36:30.254479 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.254441    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:30.254479 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.254477    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c6242d-b3eb-4131-9816-917e2951b31e-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:30.254678 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.254493    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rq78\" (UniqueName: \"kubernetes.io/projected/78c6242d-b3eb-4131-9816-917e2951b31e-kube-api-access-4rq78\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:30.254678 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.254509    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c6242d-b3eb-4131-9816-917e2951b31e-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:30.864075 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.864036    2586 generic.go:356] "Generic (PLEG): container finished" podID="78c6242d-b3eb-4131-9816-917e2951b31e" containerID="e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1" exitCode=0
Sep 09 10:36:30.864075 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.864075    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dff8d754b-k76vm"
Sep 09 10:36:30.864595 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.864125    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dff8d754b-k76vm" event={"ID":"78c6242d-b3eb-4131-9816-917e2951b31e","Type":"ContainerDied","Data":"e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1"}
Sep 09 10:36:30.864595 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.864188    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dff8d754b-k76vm" event={"ID":"78c6242d-b3eb-4131-9816-917e2951b31e","Type":"ContainerDied","Data":"33d80356555592c9113e2000e558ec6a200401740e89f81b12ac375eb5ce9466"}
Sep 09 10:36:30.864595 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.864221    2586 scope.go:117] "RemoveContainer" containerID="e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1"
Sep 09 10:36:30.875042 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.874918    2586 scope.go:117] "RemoveContainer" containerID="e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1"
Sep 09 10:36:30.875305 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:36:30.875279    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1\": container with ID starting with e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1 not found: ID does not exist" containerID="e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1"
Sep 09 10:36:30.875443 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.875315    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1"} err="failed to get container status \"e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1\": rpc error: code = NotFound desc = could not find container \"e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1\": container with ID starting with e20aa56d0ca3fdcadba323b06b01f50064214f5a01a2e87f57096ad7f5cb53c1 not found: ID does not exist"
Sep 09 10:36:30.884396 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.884370    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dff8d754b-k76vm"]
Sep 09 10:36:30.893244 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:30.893218    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dff8d754b-k76vm"]
Sep 09 10:36:31.870831 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:31.870787    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn" event={"ID":"72d8059e-141a-46a3-ac60-7f7b45fcf36d","Type":"ContainerStarted","Data":"be924a015acde97bde8c8613ad724da62373df349dba942faa2e805aff1060b2"}
Sep 09 10:36:31.887471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:31.887413    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gxvwn" podStartSLOduration=0.959576391 podStartE2EDuration="3.887396261s" totalImagesPullingTime="2.92781987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:36:31.886040443 +0000 UTC m=+633.941599493" watchObservedRunningTime="2026-09-09 10:36:31.887396261 +0000 UTC m=+633.942955310"
Sep 09 10:36:32.474437 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:32.474401    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c6242d-b3eb-4131-9816-917e2951b31e" path="/var/lib/kubelet/pods/78c6242d-b3eb-4131-9816-917e2951b31e/volumes"
Sep 09 10:36:34.312953 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.309641    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"]
Sep 09 10:36:34.312953 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.310483    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c6242d-b3eb-4131-9816-917e2951b31e" containerName="console"
Sep 09 10:36:34.312953 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.310505    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c6242d-b3eb-4131-9816-917e2951b31e" containerName="console"
Sep 09 10:36:34.312953 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.310690    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c6242d-b3eb-4131-9816-917e2951b31e" containerName="console"
Sep 09 10:36:34.315481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.315449    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.317865 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.317834    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 10:36:34.318023 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.317834    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 10:36:34.318204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.318186    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bccwt\""
Sep 09 10:36:34.320434 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.320412    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"]
Sep 09 10:36:34.384381 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.384346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.384594 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.384469    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvn6k\" (UniqueName: \"kubernetes.io/projected/7aa27215-c4a2-41a2-94fa-feef65396781-kube-api-access-nvn6k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.384594 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.384512    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.485213 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.485181    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.485476 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.485456    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvn6k\" (UniqueName: \"kubernetes.io/projected/7aa27215-c4a2-41a2-94fa-feef65396781-kube-api-access-nvn6k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.485586 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.485572    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.485677 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.485647    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.486022 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.485999    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.497813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.497784    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvn6k\" (UniqueName: \"kubernetes.io/projected/7aa27215-c4a2-41a2-94fa-feef65396781-kube-api-access-nvn6k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.625839 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.625809    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:34.898042 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:34.898018    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"]
Sep 09 10:36:34.900641 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:36:34.900605    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa27215_c4a2_41a2_94fa_feef65396781.slice/crio-13df0a026ee419f23c68be47ab33264d2eb3db70d15060cdbf398fb506128ec0 WatchSource:0}: Error finding container 13df0a026ee419f23c68be47ab33264d2eb3db70d15060cdbf398fb506128ec0: Status 404 returned error can't find the container with id 13df0a026ee419f23c68be47ab33264d2eb3db70d15060cdbf398fb506128ec0
Sep 09 10:36:35.891763 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:35.891732    2586 generic.go:356] "Generic (PLEG): container finished" podID="7aa27215-c4a2-41a2-94fa-feef65396781" containerID="912e9a4b68edcb3e4948345def27b6ef2c4234ff73c3021bb9cf2c8b3f92c188" exitCode=0
Sep 09 10:36:35.892154 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:35.891816    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz" event={"ID":"7aa27215-c4a2-41a2-94fa-feef65396781","Type":"ContainerDied","Data":"912e9a4b68edcb3e4948345def27b6ef2c4234ff73c3021bb9cf2c8b3f92c188"}
Sep 09 10:36:35.892154 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:35.891857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz" event={"ID":"7aa27215-c4a2-41a2-94fa-feef65396781","Type":"ContainerStarted","Data":"13df0a026ee419f23c68be47ab33264d2eb3db70d15060cdbf398fb506128ec0"}
Sep 09 10:36:37.903275 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:37.903241    2586 generic.go:356] "Generic (PLEG): container finished" podID="7aa27215-c4a2-41a2-94fa-feef65396781" containerID="9afe27dfe449120458338ec97784b1cb7bf18e62826dd7ab8ef9a37b37bc1b9c" exitCode=0
Sep 09 10:36:37.903620 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:37.903297    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz" event={"ID":"7aa27215-c4a2-41a2-94fa-feef65396781","Type":"ContainerDied","Data":"9afe27dfe449120458338ec97784b1cb7bf18e62826dd7ab8ef9a37b37bc1b9c"}
Sep 09 10:36:38.910124 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:38.910091    2586 generic.go:356] "Generic (PLEG): container finished" podID="7aa27215-c4a2-41a2-94fa-feef65396781" containerID="8015a3e3c67deb9f4c50e5097aaa7beeab52b45aa40e35c9f6d1d21698dfc81a" exitCode=0
Sep 09 10:36:38.910502 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:38.910179    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz" event={"ID":"7aa27215-c4a2-41a2-94fa-feef65396781","Type":"ContainerDied","Data":"8015a3e3c67deb9f4c50e5097aaa7beeab52b45aa40e35c9f6d1d21698dfc81a"}
Sep 09 10:36:40.045753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.045728    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:40.141119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.141076    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvn6k\" (UniqueName: \"kubernetes.io/projected/7aa27215-c4a2-41a2-94fa-feef65396781-kube-api-access-nvn6k\") pod \"7aa27215-c4a2-41a2-94fa-feef65396781\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") "
Sep 09 10:36:40.141334 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.141144    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-bundle\") pod \"7aa27215-c4a2-41a2-94fa-feef65396781\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") "
Sep 09 10:36:40.141334 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.141169    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-util\") pod \"7aa27215-c4a2-41a2-94fa-feef65396781\" (UID: \"7aa27215-c4a2-41a2-94fa-feef65396781\") "
Sep 09 10:36:40.141918 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.141868    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-bundle" (OuterVolumeSpecName: "bundle") pod "7aa27215-c4a2-41a2-94fa-feef65396781" (UID: "7aa27215-c4a2-41a2-94fa-feef65396781"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:36:40.143189 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.143160    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa27215-c4a2-41a2-94fa-feef65396781-kube-api-access-nvn6k" (OuterVolumeSpecName: "kube-api-access-nvn6k") pod "7aa27215-c4a2-41a2-94fa-feef65396781" (UID: "7aa27215-c4a2-41a2-94fa-feef65396781"). InnerVolumeSpecName "kube-api-access-nvn6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:36:40.241974 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.241881    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvn6k\" (UniqueName: \"kubernetes.io/projected/7aa27215-c4a2-41a2-94fa-feef65396781-kube-api-access-nvn6k\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:40.241974 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.241926    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:40.423509 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.423466    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-util" (OuterVolumeSpecName: "util") pod "7aa27215-c4a2-41a2-94fa-feef65396781" (UID: "7aa27215-c4a2-41a2-94fa-feef65396781"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:36:40.444458 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.444428    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa27215-c4a2-41a2-94fa-feef65396781-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:40.922181 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.922141    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz" event={"ID":"7aa27215-c4a2-41a2-94fa-feef65396781","Type":"ContainerDied","Data":"13df0a026ee419f23c68be47ab33264d2eb3db70d15060cdbf398fb506128ec0"}
Sep 09 10:36:40.922181 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.922184    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4r9bz"
Sep 09 10:36:40.922385 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:40.922197    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13df0a026ee419f23c68be47ab33264d2eb3db70d15060cdbf398fb506128ec0"
Sep 09 10:36:51.332345 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.332303    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"]
Sep 09 10:36:51.332753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.332663    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa27215-c4a2-41a2-94fa-feef65396781" containerName="util"
Sep 09 10:36:51.332753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.332674    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa27215-c4a2-41a2-94fa-feef65396781" containerName="util"
Sep 09 10:36:51.332753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.332700    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa27215-c4a2-41a2-94fa-feef65396781" containerName="pull"
Sep 09 10:36:51.332753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.332705    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa27215-c4a2-41a2-94fa-feef65396781" containerName="pull"
Sep 09 10:36:51.332753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.332711    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa27215-c4a2-41a2-94fa-feef65396781" containerName="extract"
Sep 09 10:36:51.332753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.332716    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa27215-c4a2-41a2-94fa-feef65396781" containerName="extract"
Sep 09 10:36:51.332964 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.332766    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa27215-c4a2-41a2-94fa-feef65396781" containerName="extract"
Sep 09 10:36:51.366652 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.366622    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"]
Sep 09 10:36:51.366785 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.366759    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.369390 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.369366    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 10:36:51.369527 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.369452    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 10:36:51.369975 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.369961    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bccwt\""
Sep 09 10:36:51.544143 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.544106    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.544323 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.544159    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvrdq\" (UniqueName: \"kubernetes.io/projected/5e05cd31-df9e-458f-9ec9-29bf99791e71-kube-api-access-kvrdq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.544323 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.544236    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.596430 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.596314    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"]
Sep 09 10:36:51.617948 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.617913    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"]
Sep 09 10:36:51.618112 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.617997    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.620563 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.620524    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-llqpt\""
Sep 09 10:36:51.620563 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.620561    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 10:36:51.620863 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.620847    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 10:36:51.621460 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.621441    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 10:36:51.623377 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.623362    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 10:36:51.644653 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.644627    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrdq\" (UniqueName: \"kubernetes.io/projected/5e05cd31-df9e-458f-9ec9-29bf99791e71-kube-api-access-kvrdq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.644762 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.644674    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.644762 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.644743    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.645083 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.645066    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.645127 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.645083    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.655305 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.655285    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrdq\" (UniqueName: \"kubernetes.io/projected/5e05cd31-df9e-458f-9ec9-29bf99791e71-kube-api-access-kvrdq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.675680 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.675658    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:51.745794 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.745762    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjsh\" (UniqueName: \"kubernetes.io/projected/eb1d453d-cab6-49d2-8a77-e7587479725a-kube-api-access-sqjsh\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.745967 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.745825    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb1d453d-cab6-49d2-8a77-e7587479725a-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.745967 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.745855    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb1d453d-cab6-49d2-8a77-e7587479725a-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.745967 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.745909    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1d453d-cab6-49d2-8a77-e7587479725a-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.746224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.746019    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb1d453d-cab6-49d2-8a77-e7587479725a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.846794 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.846706    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb1d453d-cab6-49d2-8a77-e7587479725a-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.846794 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.846747    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb1d453d-cab6-49d2-8a77-e7587479725a-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.846794 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.846772    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1d453d-cab6-49d2-8a77-e7587479725a-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.847092 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.846818    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb1d453d-cab6-49d2-8a77-e7587479725a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.847092 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.846878    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjsh\" (UniqueName: \"kubernetes.io/projected/eb1d453d-cab6-49d2-8a77-e7587479725a-kube-api-access-sqjsh\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.847192 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.847164    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb1d453d-cab6-49d2-8a77-e7587479725a-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.849254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.849228    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1d453d-cab6-49d2-8a77-e7587479725a-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.849371 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.849353    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb1d453d-cab6-49d2-8a77-e7587479725a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.849458 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.849440    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb1d453d-cab6-49d2-8a77-e7587479725a-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.866749 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.866717    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjsh\" (UniqueName: \"kubernetes.io/projected/eb1d453d-cab6-49d2-8a77-e7587479725a-kube-api-access-sqjsh\") pod \"opendatahub-operator-controller-manager-544cbf49fb-q4m59\" (UID: \"eb1d453d-cab6-49d2-8a77-e7587479725a\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.928885 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.928848    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:51.974370 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:51.974336    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"]
Sep 09 10:36:51.977654 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:36:51.977627    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e05cd31_df9e_458f_9ec9_29bf99791e71.slice/crio-c2c36a958e8e7c5d5f46d4bcdfc84cf1e1c43dcab1810009554cff9591a0ec6b WatchSource:0}: Error finding container c2c36a958e8e7c5d5f46d4bcdfc84cf1e1c43dcab1810009554cff9591a0ec6b: Status 404 returned error can't find the container with id c2c36a958e8e7c5d5f46d4bcdfc84cf1e1c43dcab1810009554cff9591a0ec6b
Sep 09 10:36:52.237703 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:52.237676    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"]
Sep 09 10:36:52.240005 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:36:52.239977    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1d453d_cab6_49d2_8a77_e7587479725a.slice/crio-12c85ead964682084e9bb7ab353a911bc55e2a721b3ae2394f2e8b47989d25cd WatchSource:0}: Error finding container 12c85ead964682084e9bb7ab353a911bc55e2a721b3ae2394f2e8b47989d25cd: Status 404 returned error can't find the container with id 12c85ead964682084e9bb7ab353a911bc55e2a721b3ae2394f2e8b47989d25cd
Sep 09 10:36:52.981806 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:52.981770    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59" event={"ID":"eb1d453d-cab6-49d2-8a77-e7587479725a","Type":"ContainerStarted","Data":"12c85ead964682084e9bb7ab353a911bc55e2a721b3ae2394f2e8b47989d25cd"}
Sep 09 10:36:52.983322 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:52.983288    2586 generic.go:356] "Generic (PLEG): container finished" podID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerID="1446d66fc0aff31b892fe91870936ef307bfaa8327595c149878db1943032563" exitCode=0
Sep 09 10:36:52.983468 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:52.983360    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj" event={"ID":"5e05cd31-df9e-458f-9ec9-29bf99791e71","Type":"ContainerDied","Data":"1446d66fc0aff31b892fe91870936ef307bfaa8327595c149878db1943032563"}
Sep 09 10:36:52.983468 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:52.983392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj" event={"ID":"5e05cd31-df9e-458f-9ec9-29bf99791e71","Type":"ContainerStarted","Data":"c2c36a958e8e7c5d5f46d4bcdfc84cf1e1c43dcab1810009554cff9591a0ec6b"}
Sep 09 10:36:53.992238 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:53.992156    2586 generic.go:356] "Generic (PLEG): container finished" podID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerID="564a8616433064013e88320ee8204c14beddc461e4cf791c301ea9360059add4" exitCode=0
Sep 09 10:36:53.992691 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:53.992284    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj" event={"ID":"5e05cd31-df9e-458f-9ec9-29bf99791e71","Type":"ContainerDied","Data":"564a8616433064013e88320ee8204c14beddc461e4cf791c301ea9360059add4"}
Sep 09 10:36:55.000245 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:55.000208    2586 generic.go:356] "Generic (PLEG): container finished" podID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerID="c751fdf71affccdf5d8692a01ef2508e84f847755dfffd48beed29d514689493" exitCode=0
Sep 09 10:36:55.000667 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:55.000299    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj" event={"ID":"5e05cd31-df9e-458f-9ec9-29bf99791e71","Type":"ContainerDied","Data":"c751fdf71affccdf5d8692a01ef2508e84f847755dfffd48beed29d514689493"}
Sep 09 10:36:56.142569 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.142543    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:56.183030 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.183005    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-bundle\") pod \"5e05cd31-df9e-458f-9ec9-29bf99791e71\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") "
Sep 09 10:36:56.183166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.183053    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvrdq\" (UniqueName: \"kubernetes.io/projected/5e05cd31-df9e-458f-9ec9-29bf99791e71-kube-api-access-kvrdq\") pod \"5e05cd31-df9e-458f-9ec9-29bf99791e71\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") "
Sep 09 10:36:56.183166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.183085    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-util\") pod \"5e05cd31-df9e-458f-9ec9-29bf99791e71\" (UID: \"5e05cd31-df9e-458f-9ec9-29bf99791e71\") "
Sep 09 10:36:56.183946 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.183878    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-bundle" (OuterVolumeSpecName: "bundle") pod "5e05cd31-df9e-458f-9ec9-29bf99791e71" (UID: "5e05cd31-df9e-458f-9ec9-29bf99791e71"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:36:56.185835 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.185805    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e05cd31-df9e-458f-9ec9-29bf99791e71-kube-api-access-kvrdq" (OuterVolumeSpecName: "kube-api-access-kvrdq") pod "5e05cd31-df9e-458f-9ec9-29bf99791e71" (UID: "5e05cd31-df9e-458f-9ec9-29bf99791e71"). InnerVolumeSpecName "kube-api-access-kvrdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:36:56.188933 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.188878    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-util" (OuterVolumeSpecName: "util") pod "5e05cd31-df9e-458f-9ec9-29bf99791e71" (UID: "5e05cd31-df9e-458f-9ec9-29bf99791e71"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:36:56.284577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.284465    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvrdq\" (UniqueName: \"kubernetes.io/projected/5e05cd31-df9e-458f-9ec9-29bf99791e71-kube-api-access-kvrdq\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:56.284577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.284515    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:56.284577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:56.284525    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e05cd31-df9e-458f-9ec9-29bf99791e71-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:36:57.014734 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:57.014696    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59" event={"ID":"eb1d453d-cab6-49d2-8a77-e7587479725a","Type":"ContainerStarted","Data":"04bfb0e444cf597de434ab94d7d10fec054f4ceb1d24057d8bbeffb01f7d86a8"}
Sep 09 10:36:57.016714 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:57.016693    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj"
Sep 09 10:36:57.016714 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:57.016705    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06v8tj" event={"ID":"5e05cd31-df9e-458f-9ec9-29bf99791e71","Type":"ContainerDied","Data":"c2c36a958e8e7c5d5f46d4bcdfc84cf1e1c43dcab1810009554cff9591a0ec6b"}
Sep 09 10:36:57.016850 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:57.016742    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c36a958e8e7c5d5f46d4bcdfc84cf1e1c43dcab1810009554cff9591a0ec6b"
Sep 09 10:36:58.023811 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:58.023776    2586 generic.go:356] "Generic (PLEG): container finished" podID="eb1d453d-cab6-49d2-8a77-e7587479725a" containerID="04bfb0e444cf597de434ab94d7d10fec054f4ceb1d24057d8bbeffb01f7d86a8" exitCode=0
Sep 09 10:36:58.024184 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:58.023831    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59" event={"ID":"eb1d453d-cab6-49d2-8a77-e7587479725a","Type":"ContainerDied","Data":"04bfb0e444cf597de434ab94d7d10fec054f4ceb1d24057d8bbeffb01f7d86a8"}
Sep 09 10:36:59.031302 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:59.031263    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59" event={"ID":"eb1d453d-cab6-49d2-8a77-e7587479725a","Type":"ContainerStarted","Data":"be190f451ff1422c3d34a14e8d1ea005c9dc6d0c1b823cbbe02b570a176eaa6a"}
Sep 09 10:36:59.031784 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:59.031394    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:36:59.063721 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:36:59.063665    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59" podStartSLOduration=2.78207264 podStartE2EDuration="8.063645886s" totalImagesPullingTime="4.252287621s" totalInitContainerRuntime="1.029285625s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:36:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:36:59.061570505 +0000 UTC m=+661.117129566" watchObservedRunningTime="2026-09-09 10:36:59.063645886 +0000 UTC m=+661.119204933"
Sep 09 10:37:10.038063 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:10.038032    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-q4m59"
Sep 09 10:37:12.319391 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.319346    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"]
Sep 09 10:37:12.319747 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.319693    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerName="extract"
Sep 09 10:37:12.319747 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.319705    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerName="extract"
Sep 09 10:37:12.319747 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.319719    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerName="pull"
Sep 09 10:37:12.319747 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.319725    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerName="pull"
Sep 09 10:37:12.319747 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.319734    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerName="util"
Sep 09 10:37:12.319747 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.319739    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerName="util"
Sep 09 10:37:12.319955 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.319788    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e05cd31-df9e-458f-9ec9-29bf99791e71" containerName="extract"
Sep 09 10:37:12.328139 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.328121    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.333658 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.333633    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 10:37:12.333802 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.333633    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-489j2\""
Sep 09 10:37:12.333802 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.333639    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 10:37:12.334292 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.334274    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 10:37:12.338541 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.338523    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 10:37:12.370506 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.370481    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"]
Sep 09 10:37:12.427839 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.427810    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a22e8b-0b98-44aa-87c7-0dc20095f080-tls-certs\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.427998 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.427853    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a22e8b-0b98-44aa-87c7-0dc20095f080-tmp\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.427998 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.427878    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbt7\" (UniqueName: \"kubernetes.io/projected/06a22e8b-0b98-44aa-87c7-0dc20095f080-kube-api-access-xhbt7\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.528875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.528843    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a22e8b-0b98-44aa-87c7-0dc20095f080-tmp\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.529065 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.528885    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbt7\" (UniqueName: \"kubernetes.io/projected/06a22e8b-0b98-44aa-87c7-0dc20095f080-kube-api-access-xhbt7\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.529065 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.528985    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a22e8b-0b98-44aa-87c7-0dc20095f080-tls-certs\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.531198 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.531174    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a22e8b-0b98-44aa-87c7-0dc20095f080-tmp\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.531297 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.531272    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a22e8b-0b98-44aa-87c7-0dc20095f080-tls-certs\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.542531 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.542506    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbt7\" (UniqueName: \"kubernetes.io/projected/06a22e8b-0b98-44aa-87c7-0dc20095f080-kube-api-access-xhbt7\") pod \"kube-auth-proxy-75cd99c5fc-84px9\" (UID: \"06a22e8b-0b98-44aa-87c7-0dc20095f080\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.637642 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.637614    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"
Sep 09 10:37:12.911255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.911227    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9"]
Sep 09 10:37:12.913382 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:37:12.913352    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a22e8b_0b98_44aa_87c7_0dc20095f080.slice/crio-c4dc6e0d629abfc47c50b867cd24715902a056eb4869c085ee56a513d424d871 WatchSource:0}: Error finding container c4dc6e0d629abfc47c50b867cd24715902a056eb4869c085ee56a513d424d871: Status 404 returned error can't find the container with id c4dc6e0d629abfc47c50b867cd24715902a056eb4869c085ee56a513d424d871
Sep 09 10:37:12.915251 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:12.915232    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:37:13.098588 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:13.098548    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9" event={"ID":"06a22e8b-0b98-44aa-87c7-0dc20095f080","Type":"ContainerStarted","Data":"c4dc6e0d629abfc47c50b867cd24715902a056eb4869c085ee56a513d424d871"}
Sep 09 10:37:16.115865 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:16.115831    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9" event={"ID":"06a22e8b-0b98-44aa-87c7-0dc20095f080","Type":"ContainerStarted","Data":"e85d82f5caa26c50dc3a0350d7d2a722a55b53f1c0d0448d8aadac5f27a687f9"}
Sep 09 10:37:16.133204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:16.133141    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-84px9" podStartSLOduration=1.229434577 podStartE2EDuration="4.133125021s" totalImagesPullingTime="2.903690444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:16.132676505 +0000 UTC m=+678.188235555" watchObservedRunningTime="2026-09-09 10:37:16.133125021 +0000 UTC m=+678.188684072"
Sep 09 10:37:18.170588 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.170554    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"]
Sep 09 10:37:18.174139 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.174114    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.176621 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.176598    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 10:37:18.176747 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.176630    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sl694\""
Sep 09 10:37:18.192082 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.192057    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"]
Sep 09 10:37:18.281397 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.281367    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"]
Sep 09 10:37:18.281766 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.281744    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.281825 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.281785    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.281879 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.281838    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.281879 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.281864    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.281982 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.281928    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.281982 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.281971    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.282046 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.281995    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.282080 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.282059    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.282121 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.282094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ww2\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-kube-api-access-b4ww2\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.284824 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.284810    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.302807 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.302781    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"]
Sep 09 10:37:18.383239 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383197    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf219d22-e6ba-474a-879e-79b963df8772-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.383442 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383264    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.383442 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383312    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383442 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383349    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383442 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383370    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf219d22-e6ba-474a-879e-79b963df8772-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.383442 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383410    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzdb\" (UniqueName: \"kubernetes.io/projected/bf219d22-e6ba-474a-879e-79b963df8772-kube-api-access-hqzdb\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383466    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383502    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383530    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383554    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383584    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383610    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383625    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383645    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383682    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf219d22-e6ba-474a-879e-79b963df8772-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383717    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.383796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383794    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.384764 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383723    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ww2\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-kube-api-access-b4ww2\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.384764 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.384764 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.383972    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.384764 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.384066    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.384764 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.384157    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.386233 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.386214    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.386539 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.386518    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.392898 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.392866    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.393079 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.393055    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ww2\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-kube-api-access-b4ww2\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bchsl4t\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.485096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485029    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:18.485211 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485156    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485211 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485211 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485206    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485400 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485240    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf219d22-e6ba-474a-879e-79b963df8772-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485400 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485270    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485400 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf219d22-e6ba-474a-879e-79b963df8772-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485400 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485344    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485400 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485377    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf219d22-e6ba-474a-879e-79b963df8772-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485420    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzdb\" (UniqueName: \"kubernetes.io/projected/bf219d22-e6ba-474a-879e-79b963df8772-kube-api-access-hqzdb\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485519    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485610    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485685    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.485818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.485763    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.486393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.486368    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf219d22-e6ba-474a-879e-79b963df8772-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.487383 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.487358    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf219d22-e6ba-474a-879e-79b963df8772-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.487989 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.487966    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf219d22-e6ba-474a-879e-79b963df8772-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.493952 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.493932    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf219d22-e6ba-474a-879e-79b963df8772-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.494225 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.494204    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzdb\" (UniqueName: \"kubernetes.io/projected/bf219d22-e6ba-474a-879e-79b963df8772-kube-api-access-hqzdb\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb\" (UID: \"bf219d22-e6ba-474a-879e-79b963df8772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.594403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.594376    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:18.770343 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.770319    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"]
Sep 09 10:37:18.772115 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:37:18.772085    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ac1b9b_77f9_4458_b2d1_4dc4a2b9b3c8.slice/crio-7df9d85c657f8016902930643bedd550f8707f0786f499ed87d438565fce147e WatchSource:0}: Error finding container 7df9d85c657f8016902930643bedd550f8707f0786f499ed87d438565fce147e: Status 404 returned error can't find the container with id 7df9d85c657f8016902930643bedd550f8707f0786f499ed87d438565fce147e
Sep 09 10:37:18.869794 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:18.869767    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"]
Sep 09 10:37:18.871521 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:37:18.871494    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf219d22_e6ba_474a_879e_79b963df8772.slice/crio-2d7b2fddba4ec35232568a67fb918c28a8f183c6d6a976ecdca4842ddd4f76f9 WatchSource:0}: Error finding container 2d7b2fddba4ec35232568a67fb918c28a8f183c6d6a976ecdca4842ddd4f76f9: Status 404 returned error can't find the container with id 2d7b2fddba4ec35232568a67fb918c28a8f183c6d6a976ecdca4842ddd4f76f9
Sep 09 10:37:19.130729 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:19.130688    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb" event={"ID":"bf219d22-e6ba-474a-879e-79b963df8772","Type":"ContainerStarted","Data":"2d7b2fddba4ec35232568a67fb918c28a8f183c6d6a976ecdca4842ddd4f76f9"}
Sep 09 10:37:19.131848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:19.131826    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" event={"ID":"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8","Type":"ContainerStarted","Data":"7df9d85c657f8016902930643bedd550f8707f0786f499ed87d438565fce147e"}
Sep 09 10:37:21.585696 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:21.585653    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:37:21.586003 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:21.585728    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:37:21.586003 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:21.585756    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:37:21.591341 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:21.591313    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:37:21.591403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:21.591386    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:37:21.591452 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:21.591437    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:37:22.151585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.151543    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb" event={"ID":"bf219d22-e6ba-474a-879e-79b963df8772","Type":"ContainerStarted","Data":"303da649512579ff9cef608983ce92c4c820660662dd55851eb46c0817e991ae"}
Sep 09 10:37:22.153347 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.153314    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" event={"ID":"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8","Type":"ContainerStarted","Data":"2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0"}
Sep 09 10:37:22.178064 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.178015    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb" podStartSLOduration=1.460999768 podStartE2EDuration="4.178000467s" totalImagesPullingTime="2.717000699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:22.176768879 +0000 UTC m=+684.232327946" watchObservedRunningTime="2026-09-09 10:37:22.178000467 +0000 UTC m=+684.233559517"
Sep 09 10:37:22.199322 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.199265    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" podStartSLOduration=1.387993126 podStartE2EDuration="4.199242682s" totalImagesPullingTime="2.811249556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:22.198975484 +0000 UTC m=+684.254534536" watchObservedRunningTime="2026-09-09 10:37:22.199242682 +0000 UTC m=+684.254801753"
Sep 09 10:37:22.486084 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.486001    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:22.487417 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.487388    2586 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 09 10:37:22.487542 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.487437    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" podUID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 09 10:37:22.595096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.595065    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:22.600048 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.600025    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:22.623323 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.623295    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4rlxq"]
Sep 09 10:37:22.627163 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.627141    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:22.631005 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.630979    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-52xh4\""
Sep 09 10:37:22.631235 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.631216    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 10:37:22.643621 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.643596    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4rlxq"]
Sep 09 10:37:22.736732 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.736635    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/107da22e-8ba0-4b76-b5a1-b1b37acd952a-cert\") pod \"odh-model-controller-7998c67dc4-4rlxq\" (UID: \"107da22e-8ba0-4b76-b5a1-b1b37acd952a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:22.736916 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.736751    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8p8\" (UniqueName: \"kubernetes.io/projected/107da22e-8ba0-4b76-b5a1-b1b37acd952a-kube-api-access-gt8p8\") pod \"odh-model-controller-7998c67dc4-4rlxq\" (UID: \"107da22e-8ba0-4b76-b5a1-b1b37acd952a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:22.837876 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.837831    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/107da22e-8ba0-4b76-b5a1-b1b37acd952a-cert\") pod \"odh-model-controller-7998c67dc4-4rlxq\" (UID: \"107da22e-8ba0-4b76-b5a1-b1b37acd952a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:22.838040 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.837932    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8p8\" (UniqueName: \"kubernetes.io/projected/107da22e-8ba0-4b76-b5a1-b1b37acd952a-kube-api-access-gt8p8\") pod \"odh-model-controller-7998c67dc4-4rlxq\" (UID: \"107da22e-8ba0-4b76-b5a1-b1b37acd952a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:22.838107 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:37:22.838034    2586 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 10:37:22.838144 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:37:22.838123    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107da22e-8ba0-4b76-b5a1-b1b37acd952a-cert podName:107da22e-8ba0-4b76-b5a1-b1b37acd952a nodeName:}" failed. No retries permitted until 2026-09-09 10:37:23.338101544 +0000 UTC m=+685.393660596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/107da22e-8ba0-4b76-b5a1-b1b37acd952a-cert") pod "odh-model-controller-7998c67dc4-4rlxq" (UID: "107da22e-8ba0-4b76-b5a1-b1b37acd952a") : secret "odh-model-controller-webhook-cert" not found
Sep 09 10:37:22.850249 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:22.850212    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8p8\" (UniqueName: \"kubernetes.io/projected/107da22e-8ba0-4b76-b5a1-b1b37acd952a-kube-api-access-gt8p8\") pod \"odh-model-controller-7998c67dc4-4rlxq\" (UID: \"107da22e-8ba0-4b76-b5a1-b1b37acd952a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:23.159036 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.159000    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:23.159880 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.159860    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb"
Sep 09 10:37:23.213677 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.213645    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xrhq2"]
Sep 09 10:37:23.217006 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.216989    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:23.219009 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.218982    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 09 10:37:23.231403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.231378    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xrhq2"]
Sep 09 10:37:23.234260 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.234239    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"]
Sep 09 10:37:23.344226 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.344188    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkt5d\" (UniqueName: \"kubernetes.io/projected/c3c2bbd0-0d37-446b-a81d-023f3132c893-kube-api-access-jkt5d\") pod \"model-serving-api-54d89bd5fc-xrhq2\" (UID: \"c3c2bbd0-0d37-446b-a81d-023f3132c893\") " pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:23.344430 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.344265    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c2bbd0-0d37-446b-a81d-023f3132c893-tls-certs\") pod \"model-serving-api-54d89bd5fc-xrhq2\" (UID: \"c3c2bbd0-0d37-446b-a81d-023f3132c893\") " pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:23.344430 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.344390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/107da22e-8ba0-4b76-b5a1-b1b37acd952a-cert\") pod \"odh-model-controller-7998c67dc4-4rlxq\" (UID: \"107da22e-8ba0-4b76-b5a1-b1b37acd952a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:23.347158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.347133    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/107da22e-8ba0-4b76-b5a1-b1b37acd952a-cert\") pod \"odh-model-controller-7998c67dc4-4rlxq\" (UID: \"107da22e-8ba0-4b76-b5a1-b1b37acd952a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:23.445375 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.445287    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkt5d\" (UniqueName: \"kubernetes.io/projected/c3c2bbd0-0d37-446b-a81d-023f3132c893-kube-api-access-jkt5d\") pod \"model-serving-api-54d89bd5fc-xrhq2\" (UID: \"c3c2bbd0-0d37-446b-a81d-023f3132c893\") " pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:23.445375 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.445348    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c2bbd0-0d37-446b-a81d-023f3132c893-tls-certs\") pod \"model-serving-api-54d89bd5fc-xrhq2\" (UID: \"c3c2bbd0-0d37-446b-a81d-023f3132c893\") " pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:23.447734 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.447706    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c2bbd0-0d37-446b-a81d-023f3132c893-tls-certs\") pod \"model-serving-api-54d89bd5fc-xrhq2\" (UID: \"c3c2bbd0-0d37-446b-a81d-023f3132c893\") " pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:23.455820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.455794    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkt5d\" (UniqueName: \"kubernetes.io/projected/c3c2bbd0-0d37-446b-a81d-023f3132c893-kube-api-access-jkt5d\") pod \"model-serving-api-54d89bd5fc-xrhq2\" (UID: \"c3c2bbd0-0d37-446b-a81d-023f3132c893\") " pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:23.485863 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.485841    2586 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 09 10:37:23.485958 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.485884    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" podUID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 09 10:37:23.530347 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.530327    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:23.541401 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.541379    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:23.806820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.806794    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xrhq2"]
Sep 09 10:37:23.807969 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:37:23.807938    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c2bbd0_0d37_446b_a81d_023f3132c893.slice/crio-11ef4507b8bcac73ae6c97633269dab8e9ec7def07a819bd9957da4e30d19892 WatchSource:0}: Error finding container 11ef4507b8bcac73ae6c97633269dab8e9ec7def07a819bd9957da4e30d19892: Status 404 returned error can't find the container with id 11ef4507b8bcac73ae6c97633269dab8e9ec7def07a819bd9957da4e30d19892
Sep 09 10:37:23.817341 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:23.817317    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4rlxq"]
Sep 09 10:37:23.818643 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:37:23.818621    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107da22e_8ba0_4b76_b5a1_b1b37acd952a.slice/crio-9acadb8b37e2299b1ece3091c7fbca62ac14b67dcb8c345de779d1e7fc80f652 WatchSource:0}: Error finding container 9acadb8b37e2299b1ece3091c7fbca62ac14b67dcb8c345de779d1e7fc80f652: Status 404 returned error can't find the container with id 9acadb8b37e2299b1ece3091c7fbca62ac14b67dcb8c345de779d1e7fc80f652
Sep 09 10:37:24.165717 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:24.165672    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq" event={"ID":"107da22e-8ba0-4b76-b5a1-b1b37acd952a","Type":"ContainerStarted","Data":"9acadb8b37e2299b1ece3091c7fbca62ac14b67dcb8c345de779d1e7fc80f652"}
Sep 09 10:37:24.166920 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:24.166873    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2" event={"ID":"c3c2bbd0-0d37-446b-a81d-023f3132c893","Type":"ContainerStarted","Data":"11ef4507b8bcac73ae6c97633269dab8e9ec7def07a819bd9957da4e30d19892"}
Sep 09 10:37:24.485401 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:24.485326    2586 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 09 10:37:24.485546 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:24.485396    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" podUID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 09 10:37:25.175166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:25.175122    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" podUID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" containerName="istio-proxy" containerID="cri-o://2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0" gracePeriod=30
Sep 09 10:37:26.182817 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:26.182775    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2" event={"ID":"c3c2bbd0-0d37-446b-a81d-023f3132c893","Type":"ContainerStarted","Data":"844c073a6d184545f42a6dd95990f069ad9fa5b724d3abb53a4b3bbbba7bce83"}
Sep 09 10:37:26.183304 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:26.182930    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:26.201467 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:26.201419    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2" podStartSLOduration=1.583437279 podStartE2EDuration="3.201404583s" totalImagesPullingTime="1.617967304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:26.198389905 +0000 UTC m=+688.253948968" watchObservedRunningTime="2026-09-09 10:37:26.201404583 +0000 UTC m=+688.256963632"
Sep 09 10:37:27.188969 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:27.188929    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq" event={"ID":"107da22e-8ba0-4b76-b5a1-b1b37acd952a","Type":"ContainerStarted","Data":"1844b0cde4c72eee3ee3f029e00fe86c22171a47d4d97dee7e5c0cd8ae839f91"}
Sep 09 10:37:27.212965 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:27.212917    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq" podStartSLOduration=1.97493774 podStartE2EDuration="5.212887147s" totalImagesPullingTime="3.237949407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:27.211506072 +0000 UTC m=+689.267065133" watchObservedRunningTime="2026-09-09 10:37:27.212887147 +0000 UTC m=+689.268446246"
Sep 09 10:37:28.194672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:28.194644    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:30.437777 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.437720    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:30.619953 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.619916    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-certs\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.619983    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-token\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620017    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-socket\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620033    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4ww2\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-kube-api-access-b4ww2\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620058    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-credential-socket\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620103    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istiod-ca-cert\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620125    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-podinfo\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620149    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-data\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620193    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-envoy\") pod \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\" (UID: \"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8\") "
Sep 09 10:37:30.620481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620227    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:37:30.620481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620421    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:37:30.620481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620445    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:37:30.620481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620431    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:37:30.620771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620502    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:37:30.620771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620529    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.620771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620549    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.620771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620564    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.620771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.620581    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.622553 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.622521    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:37:30.622553 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.622536    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-kube-api-access-b4ww2" (OuterVolumeSpecName: "kube-api-access-b4ww2") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "kube-api-access-b4ww2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:37:30.622728 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.622544    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:37:30.622728 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.622552    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" (UID: "70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:37:30.638803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.638775    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"]
Sep 09 10:37:30.639223 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.639207    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" containerName="istio-proxy"
Sep 09 10:37:30.639307 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.639224    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" containerName="istio-proxy"
Sep 09 10:37:30.639369 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.639334    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" containerName="istio-proxy"
Sep 09 10:37:30.642721 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.642703    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.645109 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.645086    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ch6ct\""
Sep 09 10:37:30.645205 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.645142    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 10:37:30.645299 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.645281    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 10:37:30.645557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.645540    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 10:37:30.655393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.655370    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"]
Sep 09 10:37:30.721702 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4k5q\" (UniqueName: \"kubernetes.io/projected/7cd10489-77ac-48fe-bed6-ab99f84a2c18-kube-api-access-q4k5q\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.721702 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721670    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7cd10489-77ac-48fe-bed6-ab99f84a2c18-manager-config\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.721913 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721777    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cd10489-77ac-48fe-bed6-ab99f84a2c18-cert\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.721913 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721809    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cd10489-77ac-48fe-bed6-ab99f84a2c18-metrics-cert\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.721913 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721841    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ww2\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-kube-api-access-b4ww2\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.721913 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721866    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.721913 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721876    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.721913 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721884    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.721913 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.721906    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:37:30.822538 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.822505    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7cd10489-77ac-48fe-bed6-ab99f84a2c18-manager-config\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.822724 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.822571    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cd10489-77ac-48fe-bed6-ab99f84a2c18-cert\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.822724 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.822597    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cd10489-77ac-48fe-bed6-ab99f84a2c18-metrics-cert\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.822724 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.822623    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4k5q\" (UniqueName: \"kubernetes.io/projected/7cd10489-77ac-48fe-bed6-ab99f84a2c18-kube-api-access-q4k5q\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.823347 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.823322    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7cd10489-77ac-48fe-bed6-ab99f84a2c18-manager-config\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.825083 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.825060    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cd10489-77ac-48fe-bed6-ab99f84a2c18-cert\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.825156 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.825087    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cd10489-77ac-48fe-bed6-ab99f84a2c18-metrics-cert\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.838529 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.838509    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4k5q\" (UniqueName: \"kubernetes.io/projected/7cd10489-77ac-48fe-bed6-ab99f84a2c18-kube-api-access-q4k5q\") pod \"lws-controller-manager-7658745f98-pgwlc\" (UID: \"7cd10489-77ac-48fe-bed6-ab99f84a2c18\") " pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:30.952753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:30.952716    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:31.214481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.214451    2586 generic.go:356] "Generic (PLEG): container finished" podID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" containerID="2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0" exitCode=0
Sep 09 10:37:31.214656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.214504    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"
Sep 09 10:37:31.214656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.214530    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" event={"ID":"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8","Type":"ContainerDied","Data":"2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0"}
Sep 09 10:37:31.214656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.214589    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t" event={"ID":"70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8","Type":"ContainerDied","Data":"7df9d85c657f8016902930643bedd550f8707f0786f499ed87d438565fce147e"}
Sep 09 10:37:31.214656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.214611    2586 scope.go:117] "RemoveContainer" containerID="2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0"
Sep 09 10:37:31.224525 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.224506    2586 scope.go:117] "RemoveContainer" containerID="2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0"
Sep 09 10:37:31.224800 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:37:31.224783    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0\": container with ID starting with 2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0 not found: ID does not exist" containerID="2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0"
Sep 09 10:37:31.224847 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.224810    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0"} err="failed to get container status \"2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0\": rpc error: code = NotFound desc = could not find container \"2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0\": container with ID starting with 2ce198e1a15d41dec94fc88f6e860a75868928f6a0ef51159a82d2e46e824ce0 not found: ID does not exist"
Sep 09 10:37:31.236608 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.236586    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"]
Sep 09 10:37:31.238074 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:37:31.238052    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd10489_77ac_48fe_bed6_ab99f84a2c18.slice/crio-f493df0cc03f99cff471db5e3c09165bfcb872634b7cafc71248cfedde361ed8 WatchSource:0}: Error finding container f493df0cc03f99cff471db5e3c09165bfcb872634b7cafc71248cfedde361ed8: Status 404 returned error can't find the container with id f493df0cc03f99cff471db5e3c09165bfcb872634b7cafc71248cfedde361ed8
Sep 09 10:37:31.241364 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.241344    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"]
Sep 09 10:37:31.244020 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:31.243994    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bchsl4t"]
Sep 09 10:37:32.223460 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:32.223417    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc" event={"ID":"7cd10489-77ac-48fe-bed6-ab99f84a2c18","Type":"ContainerStarted","Data":"f493df0cc03f99cff471db5e3c09165bfcb872634b7cafc71248cfedde361ed8"}
Sep 09 10:37:32.475458 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:32.475431    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8" path="/var/lib/kubelet/pods/70ac1b9b-77f9-4458-b2d1-4dc4a2b9b3c8/volumes"
Sep 09 10:37:33.231114 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:33.231075    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc" event={"ID":"7cd10489-77ac-48fe-bed6-ab99f84a2c18","Type":"ContainerStarted","Data":"e967f0fa33e9e9fe35bd0fd459e239820a039364757830401e1dcbe7f8749435"}
Sep 09 10:37:33.231495 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:33.231217    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:37:37.193152 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:37.193121    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xrhq2"
Sep 09 10:37:37.213289 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:37.213235    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc" podStartSLOduration=5.989652701 podStartE2EDuration="7.213219389s" totalImagesPullingTime="1.223566688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:33.285717827 +0000 UTC m=+695.341276877" watchObservedRunningTime="2026-09-09 10:37:37.213219389 +0000 UTC m=+699.268778449"
Sep 09 10:37:39.202980 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:39.202950    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4rlxq"
Sep 09 10:37:44.243661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:37:44.243631    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7658745f98-pgwlc"
Sep 09 10:38:28.291366 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.291328    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"]
Sep 09 10:38:28.295306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.295286    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.297539 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.297517    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 10:38:28.298209 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.298186    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 10:38:28.298408 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.298389    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bccwt\""
Sep 09 10:38:28.304585 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.304559    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"]
Sep 09 10:38:28.355063 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.355030    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.355233 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.355078    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4fs\" (UniqueName: \"kubernetes.io/projected/8de33119-d62c-4a3a-a6ba-610cd34c0588-kube-api-access-ln4fs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.355233 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.355178    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.386213 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.386184    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"]
Sep 09 10:38:28.390368 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.390349    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.401915 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.401861    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"]
Sep 09 10:38:28.455954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.455921    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4fs\" (UniqueName: \"kubernetes.io/projected/8de33119-d62c-4a3a-a6ba-610cd34c0588-kube-api-access-ln4fs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.456183 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.455972    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5mm\" (UniqueName: \"kubernetes.io/projected/d5830d9d-6c92-4a7e-89de-118462515c5e-kube-api-access-qx5mm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.456183 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.456035    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.456183 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.456064    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.456183 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.456140    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.456183 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.456167    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.456401 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.456381    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.456456 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.456438    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.463954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.463929    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4fs\" (UniqueName: \"kubernetes.io/projected/8de33119-d62c-4a3a-a6ba-610cd34c0588-kube-api-access-ln4fs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.478942 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.478915    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"]
Sep 09 10:38:28.482789 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.482773    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.491682 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.491661    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"]
Sep 09 10:38:28.556820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.556726    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.556820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.556762    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.556820 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.556793    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.557138 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.556855    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpng\" (UniqueName: \"kubernetes.io/projected/87a96db0-ce11-477f-bc82-606b21f2b6ea-kube-api-access-8vpng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.557138 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.556962    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5mm\" (UniqueName: \"kubernetes.io/projected/d5830d9d-6c92-4a7e-89de-118462515c5e-kube-api-access-qx5mm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.557138 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.557074    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.557571 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.557550    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.557622 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.557576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.565765 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.565747    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5mm\" (UniqueName: \"kubernetes.io/projected/d5830d9d-6c92-4a7e-89de-118462515c5e-kube-api-access-qx5mm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.585157 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.585130    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"]
Sep 09 10:38:28.592548 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.592514    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.598846 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.598819    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"]
Sep 09 10:38:28.604969 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.604945    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:28.658319 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.658247    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.658319 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.658292    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvc54\" (UniqueName: \"kubernetes.io/projected/fdc12fb5-db0d-4581-a11e-4e3ade689d00-kube-api-access-bvc54\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.658463 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.658363    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.658519 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.658462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.658519 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.658487    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.658632 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.658542    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpng\" (UniqueName: \"kubernetes.io/projected/87a96db0-ce11-477f-bc82-606b21f2b6ea-kube-api-access-8vpng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.659268 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.659210    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.659394 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.659263    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.668356 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.668323    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpng\" (UniqueName: \"kubernetes.io/projected/87a96db0-ce11-477f-bc82-606b21f2b6ea-kube-api-access-8vpng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.699935 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.699902    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:28.759662 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.759629    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.759769 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.759679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvc54\" (UniqueName: \"kubernetes.io/projected/fdc12fb5-db0d-4581-a11e-4e3ade689d00-kube-api-access-bvc54\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.759769 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.759746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.760035 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.760015    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.760216 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.760196    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.769304 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.769280    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvc54\" (UniqueName: \"kubernetes.io/projected/fdc12fb5-db0d-4581-a11e-4e3ade689d00-kube-api-access-bvc54\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.793773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.793742    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:28.883098 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.883073    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"]
Sep 09 10:38:28.884151 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:38:28.884125    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de33119_d62c_4a3a_a6ba_610cd34c0588.slice/crio-e583ec3f41fb02b96ec1a366930f453fdfc2768a82f23aafb981994024ef549d WatchSource:0}: Error finding container e583ec3f41fb02b96ec1a366930f453fdfc2768a82f23aafb981994024ef549d: Status 404 returned error can't find the container with id e583ec3f41fb02b96ec1a366930f453fdfc2768a82f23aafb981994024ef549d
Sep 09 10:38:28.905648 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.905618    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:28.978158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:28.978124    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"]
Sep 09 10:38:28.978277 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:38:28.978156    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5830d9d_6c92_4a7e_89de_118462515c5e.slice/crio-4e9e884e84acdd517a214fffb897aff3566024fccf89ab26c3d78e5643fb684a WatchSource:0}: Error finding container 4e9e884e84acdd517a214fffb897aff3566024fccf89ab26c3d78e5643fb684a: Status 404 returned error can't find the container with id 4e9e884e84acdd517a214fffb897aff3566024fccf89ab26c3d78e5643fb684a
Sep 09 10:38:29.068684 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.068660    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"]
Sep 09 10:38:29.070691 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:38:29.070658    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a96db0_ce11_477f_bc82_606b21f2b6ea.slice/crio-c43063a0c71e235533b914410844a7fcfeff71a8c093e1d75b72f63bf1e1870e WatchSource:0}: Error finding container c43063a0c71e235533b914410844a7fcfeff71a8c093e1d75b72f63bf1e1870e: Status 404 returned error can't find the container with id c43063a0c71e235533b914410844a7fcfeff71a8c093e1d75b72f63bf1e1870e
Sep 09 10:38:29.183026 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.183000    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"]
Sep 09 10:38:29.231832 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:38:29.231795    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc12fb5_db0d_4581_a11e_4e3ade689d00.slice/crio-d736e925e84b51d0c56e7c1d2355695b6e283d2c6c38c4acbdbf9f146102d6a5 WatchSource:0}: Error finding container d736e925e84b51d0c56e7c1d2355695b6e283d2c6c38c4acbdbf9f146102d6a5: Status 404 returned error can't find the container with id d736e925e84b51d0c56e7c1d2355695b6e283d2c6c38c4acbdbf9f146102d6a5
Sep 09 10:38:29.526596 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.526560    2586 generic.go:356] "Generic (PLEG): container finished" podID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerID="5fa9d1468bdb2bb63b3d40c59bf307b67bc41b7a1a907f0aa3db9b0eb6048d1e" exitCode=0
Sep 09 10:38:29.527031 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.526653    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2" event={"ID":"fdc12fb5-db0d-4581-a11e-4e3ade689d00","Type":"ContainerDied","Data":"5fa9d1468bdb2bb63b3d40c59bf307b67bc41b7a1a907f0aa3db9b0eb6048d1e"}
Sep 09 10:38:29.527031 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.526700    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2" event={"ID":"fdc12fb5-db0d-4581-a11e-4e3ade689d00","Type":"ContainerStarted","Data":"d736e925e84b51d0c56e7c1d2355695b6e283d2c6c38c4acbdbf9f146102d6a5"}
Sep 09 10:38:29.528406 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.528378    2586 generic.go:356] "Generic (PLEG): container finished" podID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerID="ef73baf198a40605f1d44d8ff63307c86225aa6302a9d2a1ca9014e3832836d9" exitCode=0
Sep 09 10:38:29.528528 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.528470    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8" event={"ID":"87a96db0-ce11-477f-bc82-606b21f2b6ea","Type":"ContainerDied","Data":"ef73baf198a40605f1d44d8ff63307c86225aa6302a9d2a1ca9014e3832836d9"}
Sep 09 10:38:29.528528 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.528520    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8" event={"ID":"87a96db0-ce11-477f-bc82-606b21f2b6ea","Type":"ContainerStarted","Data":"c43063a0c71e235533b914410844a7fcfeff71a8c093e1d75b72f63bf1e1870e"}
Sep 09 10:38:29.530319 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.530297    2586 generic.go:356] "Generic (PLEG): container finished" podID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerID="e6b22438fe59455513bcb7cfab5676fe24371988481e13cde2d31b61f778c267" exitCode=0
Sep 09 10:38:29.530393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.530330    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd" event={"ID":"d5830d9d-6c92-4a7e-89de-118462515c5e","Type":"ContainerDied","Data":"e6b22438fe59455513bcb7cfab5676fe24371988481e13cde2d31b61f778c267"}
Sep 09 10:38:29.530393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.530365    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd" event={"ID":"d5830d9d-6c92-4a7e-89de-118462515c5e","Type":"ContainerStarted","Data":"4e9e884e84acdd517a214fffb897aff3566024fccf89ab26c3d78e5643fb684a"}
Sep 09 10:38:29.532153 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.532134    2586 generic.go:356] "Generic (PLEG): container finished" podID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerID="c0771bc27ee7ba851d6b13fc177b2c1ba714ff3eb9e9bb709ae3565a1cf17b7e" exitCode=0
Sep 09 10:38:29.532247 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.532204    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b" event={"ID":"8de33119-d62c-4a3a-a6ba-610cd34c0588","Type":"ContainerDied","Data":"c0771bc27ee7ba851d6b13fc177b2c1ba714ff3eb9e9bb709ae3565a1cf17b7e"}
Sep 09 10:38:29.532247 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:29.532232    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b" event={"ID":"8de33119-d62c-4a3a-a6ba-610cd34c0588","Type":"ContainerStarted","Data":"e583ec3f41fb02b96ec1a366930f453fdfc2768a82f23aafb981994024ef549d"}
Sep 09 10:38:31.545328 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:31.545287    2586 generic.go:356] "Generic (PLEG): container finished" podID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerID="9e1a2d5d36f75c713a12779aaaed3cac04ff3091bd4690b5a401637eb0a51153" exitCode=0
Sep 09 10:38:31.545809 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:31.545373    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2" event={"ID":"fdc12fb5-db0d-4581-a11e-4e3ade689d00","Type":"ContainerDied","Data":"9e1a2d5d36f75c713a12779aaaed3cac04ff3091bd4690b5a401637eb0a51153"}
Sep 09 10:38:31.547411 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:31.547389    2586 generic.go:356] "Generic (PLEG): container finished" podID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerID="1835ef3d334096d7e801d689b6f1c8cd8ee522862a2b52ab36b3c42fed9c9033" exitCode=0
Sep 09 10:38:31.547489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:31.547469    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b" event={"ID":"8de33119-d62c-4a3a-a6ba-610cd34c0588","Type":"ContainerDied","Data":"1835ef3d334096d7e801d689b6f1c8cd8ee522862a2b52ab36b3c42fed9c9033"}
Sep 09 10:38:32.554796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:32.554705    2586 generic.go:356] "Generic (PLEG): container finished" podID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerID="760ba92a7cdd9b9e2ee271cea0686badd9ab1a379860a8c882a4f1a1efcb7902" exitCode=0
Sep 09 10:38:32.555248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:32.554792    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2" event={"ID":"fdc12fb5-db0d-4581-a11e-4e3ade689d00","Type":"ContainerDied","Data":"760ba92a7cdd9b9e2ee271cea0686badd9ab1a379860a8c882a4f1a1efcb7902"}
Sep 09 10:38:32.557014 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:32.556981    2586 generic.go:356] "Generic (PLEG): container finished" podID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerID="2daddc94fa6cdf5e4c8123c1ecd217665b4b52247b3d458efc4b2441ee9cdf7e" exitCode=0
Sep 09 10:38:32.557153 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:32.557049    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b" event={"ID":"8de33119-d62c-4a3a-a6ba-610cd34c0588","Type":"ContainerDied","Data":"2daddc94fa6cdf5e4c8123c1ecd217665b4b52247b3d458efc4b2441ee9cdf7e"}
Sep 09 10:38:33.700613 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.700587    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:33.735024 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.734996    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:33.811471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.811427    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-bundle\") pod \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") "
Sep 09 10:38:33.811471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.811474    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-util\") pod \"8de33119-d62c-4a3a-a6ba-610cd34c0588\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") "
Sep 09 10:38:33.811683 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.811505    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-util\") pod \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") "
Sep 09 10:38:33.811683 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.811577    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-bundle\") pod \"8de33119-d62c-4a3a-a6ba-610cd34c0588\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") "
Sep 09 10:38:33.811683 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.811648    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln4fs\" (UniqueName: \"kubernetes.io/projected/8de33119-d62c-4a3a-a6ba-610cd34c0588-kube-api-access-ln4fs\") pod \"8de33119-d62c-4a3a-a6ba-610cd34c0588\" (UID: \"8de33119-d62c-4a3a-a6ba-610cd34c0588\") "
Sep 09 10:38:33.811683 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.811671    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvc54\" (UniqueName: \"kubernetes.io/projected/fdc12fb5-db0d-4581-a11e-4e3ade689d00-kube-api-access-bvc54\") pod \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\" (UID: \"fdc12fb5-db0d-4581-a11e-4e3ade689d00\") "
Sep 09 10:38:33.812119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.812084    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-bundle" (OuterVolumeSpecName: "bundle") pod "fdc12fb5-db0d-4581-a11e-4e3ade689d00" (UID: "fdc12fb5-db0d-4581-a11e-4e3ade689d00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:38:33.812256 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.812118    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-bundle" (OuterVolumeSpecName: "bundle") pod "8de33119-d62c-4a3a-a6ba-610cd34c0588" (UID: "8de33119-d62c-4a3a-a6ba-610cd34c0588"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:38:33.813865 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.813832    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc12fb5-db0d-4581-a11e-4e3ade689d00-kube-api-access-bvc54" (OuterVolumeSpecName: "kube-api-access-bvc54") pod "fdc12fb5-db0d-4581-a11e-4e3ade689d00" (UID: "fdc12fb5-db0d-4581-a11e-4e3ade689d00"). InnerVolumeSpecName "kube-api-access-bvc54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:38:33.814178 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.814151    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de33119-d62c-4a3a-a6ba-610cd34c0588-kube-api-access-ln4fs" (OuterVolumeSpecName: "kube-api-access-ln4fs") pod "8de33119-d62c-4a3a-a6ba-610cd34c0588" (UID: "8de33119-d62c-4a3a-a6ba-610cd34c0588"). InnerVolumeSpecName "kube-api-access-ln4fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:38:33.817467 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.817423    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-util" (OuterVolumeSpecName: "util") pod "fdc12fb5-db0d-4581-a11e-4e3ade689d00" (UID: "fdc12fb5-db0d-4581-a11e-4e3ade689d00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:38:33.818114 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.818094    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-util" (OuterVolumeSpecName: "util") pod "8de33119-d62c-4a3a-a6ba-610cd34c0588" (UID: "8de33119-d62c-4a3a-a6ba-610cd34c0588"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:38:33.913412 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.913381    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:33.913412 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.913409    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:33.913412 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.913417    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdc12fb5-db0d-4581-a11e-4e3ade689d00-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:33.913652 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.913428    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de33119-d62c-4a3a-a6ba-610cd34c0588-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:33.913652 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.913437    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln4fs\" (UniqueName: \"kubernetes.io/projected/8de33119-d62c-4a3a-a6ba-610cd34c0588-kube-api-access-ln4fs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:33.913652 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:33.913447    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvc54\" (UniqueName: \"kubernetes.io/projected/fdc12fb5-db0d-4581-a11e-4e3ade689d00-kube-api-access-bvc54\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:34.571405 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:34.571376    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2"
Sep 09 10:38:34.571591 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:34.571414    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74fkz2" event={"ID":"fdc12fb5-db0d-4581-a11e-4e3ade689d00","Type":"ContainerDied","Data":"d736e925e84b51d0c56e7c1d2355695b6e283d2c6c38c4acbdbf9f146102d6a5"}
Sep 09 10:38:34.571591 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:34.571472    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d736e925e84b51d0c56e7c1d2355695b6e283d2c6c38c4acbdbf9f146102d6a5"
Sep 09 10:38:34.573734 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:34.573709    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b" event={"ID":"8de33119-d62c-4a3a-a6ba-610cd34c0588","Type":"ContainerDied","Data":"e583ec3f41fb02b96ec1a366930f453fdfc2768a82f23aafb981994024ef549d"}
Sep 09 10:38:34.573856 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:34.573755    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e583ec3f41fb02b96ec1a366930f453fdfc2768a82f23aafb981994024ef549d"
Sep 09 10:38:34.573856 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:34.573717    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dxs4b"
Sep 09 10:38:40.610822 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:40.610785    2586 generic.go:356] "Generic (PLEG): container finished" podID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerID="503771fe88fe1b46517382498203fd13d9956c39b138bc0b5c84f4bc0b0b6db1" exitCode=0
Sep 09 10:38:40.611345 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:40.610868    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8" event={"ID":"87a96db0-ce11-477f-bc82-606b21f2b6ea","Type":"ContainerDied","Data":"503771fe88fe1b46517382498203fd13d9956c39b138bc0b5c84f4bc0b0b6db1"}
Sep 09 10:38:40.612840 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:40.612818    2586 generic.go:356] "Generic (PLEG): container finished" podID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerID="8a3cf52966aa7ee222e2db21621304e4bcbcc55326efbeccff5b8a00a8a59789" exitCode=0
Sep 09 10:38:40.612944 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:40.612867    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd" event={"ID":"d5830d9d-6c92-4a7e-89de-118462515c5e","Type":"ContainerDied","Data":"8a3cf52966aa7ee222e2db21621304e4bcbcc55326efbeccff5b8a00a8a59789"}
Sep 09 10:38:41.620651 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:41.620611    2586 generic.go:356] "Generic (PLEG): container finished" podID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerID="c2acbfdbb0e474b60447b533615401a1cd6de317ce637092e9e02b51150d3d68" exitCode=0
Sep 09 10:38:41.621119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:41.620681    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8" event={"ID":"87a96db0-ce11-477f-bc82-606b21f2b6ea","Type":"ContainerDied","Data":"c2acbfdbb0e474b60447b533615401a1cd6de317ce637092e9e02b51150d3d68"}
Sep 09 10:38:41.623033 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:41.623012    2586 generic.go:356] "Generic (PLEG): container finished" podID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerID="febe151e17b9ae315d029e138ca3b19b8a6f384646abb4ed10dc9f28a52c9471" exitCode=0
Sep 09 10:38:41.623109 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:41.623090    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd" event={"ID":"d5830d9d-6c92-4a7e-89de-118462515c5e","Type":"ContainerDied","Data":"febe151e17b9ae315d029e138ca3b19b8a6f384646abb4ed10dc9f28a52c9471"}
Sep 09 10:38:42.758320 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.758296    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:42.795504 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.795478    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:42.899331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.899229    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-util\") pod \"d5830d9d-6c92-4a7e-89de-118462515c5e\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") "
Sep 09 10:38:42.899331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.899309    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-bundle\") pod \"d5830d9d-6c92-4a7e-89de-118462515c5e\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") "
Sep 09 10:38:42.899331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.899334    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-bundle\") pod \"87a96db0-ce11-477f-bc82-606b21f2b6ea\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") "
Sep 09 10:38:42.899623 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.899351    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-util\") pod \"87a96db0-ce11-477f-bc82-606b21f2b6ea\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") "
Sep 09 10:38:42.899623 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.899382    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vpng\" (UniqueName: \"kubernetes.io/projected/87a96db0-ce11-477f-bc82-606b21f2b6ea-kube-api-access-8vpng\") pod \"87a96db0-ce11-477f-bc82-606b21f2b6ea\" (UID: \"87a96db0-ce11-477f-bc82-606b21f2b6ea\") "
Sep 09 10:38:42.899623 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.899447    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx5mm\" (UniqueName: \"kubernetes.io/projected/d5830d9d-6c92-4a7e-89de-118462515c5e-kube-api-access-qx5mm\") pod \"d5830d9d-6c92-4a7e-89de-118462515c5e\" (UID: \"d5830d9d-6c92-4a7e-89de-118462515c5e\") "
Sep 09 10:38:42.900013 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.899979    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-bundle" (OuterVolumeSpecName: "bundle") pod "87a96db0-ce11-477f-bc82-606b21f2b6ea" (UID: "87a96db0-ce11-477f-bc82-606b21f2b6ea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:38:42.900154 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.900117    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-bundle" (OuterVolumeSpecName: "bundle") pod "d5830d9d-6c92-4a7e-89de-118462515c5e" (UID: "d5830d9d-6c92-4a7e-89de-118462515c5e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:38:42.901619 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.901592    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a96db0-ce11-477f-bc82-606b21f2b6ea-kube-api-access-8vpng" (OuterVolumeSpecName: "kube-api-access-8vpng") pod "87a96db0-ce11-477f-bc82-606b21f2b6ea" (UID: "87a96db0-ce11-477f-bc82-606b21f2b6ea"). InnerVolumeSpecName "kube-api-access-8vpng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:38:42.901937 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.901917    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5830d9d-6c92-4a7e-89de-118462515c5e-kube-api-access-qx5mm" (OuterVolumeSpecName: "kube-api-access-qx5mm") pod "d5830d9d-6c92-4a7e-89de-118462515c5e" (UID: "d5830d9d-6c92-4a7e-89de-118462515c5e"). InnerVolumeSpecName "kube-api-access-qx5mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:38:42.904975 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.904952    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-util" (OuterVolumeSpecName: "util") pod "d5830d9d-6c92-4a7e-89de-118462515c5e" (UID: "d5830d9d-6c92-4a7e-89de-118462515c5e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:38:42.906167 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:42.906140    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-util" (OuterVolumeSpecName: "util") pod "87a96db0-ce11-477f-bc82-606b21f2b6ea" (UID: "87a96db0-ce11-477f-bc82-606b21f2b6ea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:38:43.000302 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.000261    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:43.000302 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.000294    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5830d9d-6c92-4a7e-89de-118462515c5e-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:43.000302 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.000303    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:43.000302 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.000317    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87a96db0-ce11-477f-bc82-606b21f2b6ea-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:43.000572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.000326    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpng\" (UniqueName: \"kubernetes.io/projected/87a96db0-ce11-477f-bc82-606b21f2b6ea-kube-api-access-8vpng\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:43.000572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.000336    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx5mm\" (UniqueName: \"kubernetes.io/projected/d5830d9d-6c92-4a7e-89de-118462515c5e-kube-api-access-qx5mm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:38:43.635632 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.635604    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8"
Sep 09 10:38:43.635814 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.635600    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjp7z8" event={"ID":"87a96db0-ce11-477f-bc82-606b21f2b6ea","Type":"ContainerDied","Data":"c43063a0c71e235533b914410844a7fcfeff71a8c093e1d75b72f63bf1e1870e"}
Sep 09 10:38:43.635814 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.635730    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c43063a0c71e235533b914410844a7fcfeff71a8c093e1d75b72f63bf1e1870e"
Sep 09 10:38:43.637787 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.637767    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd"
Sep 09 10:38:43.637923 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.637802    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4nqcd" event={"ID":"d5830d9d-6c92-4a7e-89de-118462515c5e","Type":"ContainerDied","Data":"4e9e884e84acdd517a214fffb897aff3566024fccf89ab26c3d78e5643fb684a"}
Sep 09 10:38:43.637923 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:38:43.637841    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e9e884e84acdd517a214fffb897aff3566024fccf89ab26c3d78e5643fb684a"
Sep 09 10:40:02.836848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.836804    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:40:02.837464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837404    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerName="pull"
Sep 09 10:40:02.837464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837424    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerName="pull"
Sep 09 10:40:02.837464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837452    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerName="util"
Sep 09 10:40:02.837464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837460    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerName="util"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837476    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerName="util"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837484    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerName="util"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837494    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerName="pull"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837503    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerName="pull"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837514    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerName="pull"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837522    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerName="pull"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837533    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerName="extract"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837541    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerName="extract"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837556    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerName="pull"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837565    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerName="pull"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837582    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerName="util"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837590    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerName="util"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837605    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerName="extract"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837615    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerName="extract"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837625    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerName="util"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837634    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerName="util"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837645    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerName="extract"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837654    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerName="extract"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837665    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerName="extract"
Sep 09 10:40:02.837690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837675    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerName="extract"
Sep 09 10:40:02.838623 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837784    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5830d9d-6c92-4a7e-89de-118462515c5e" containerName="extract"
Sep 09 10:40:02.838623 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837797    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a96db0-ce11-477f-bc82-606b21f2b6ea" containerName="extract"
Sep 09 10:40:02.838623 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837805    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc12fb5-db0d-4581-a11e-4e3ade689d00" containerName="extract"
Sep 09 10:40:02.838623 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.837814    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de33119-d62c-4a3a-a6ba-610cd34c0588" containerName="extract"
Sep 09 10:40:02.843077 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.843042    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:02.847924 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.847823    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 10:40:02.848493 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.848470    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 10:40:02.848627 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.848573    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m88d8\""
Sep 09 10:40:02.848718 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.848691    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 10:40:02.850908 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.850869    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:40:02.865872 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.865840    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:40:02.921461 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.921421    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vqk\" (UniqueName: \"kubernetes.io/projected/17604753-84a4-413e-aac3-3dab4dca5ae7-kube-api-access-55vqk\") pod \"limitador-limitador-84994f7ffc-w5fxt\" (UID: \"17604753-84a4-413e-aac3-3dab4dca5ae7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:02.921659 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:02.921523    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17604753-84a4-413e-aac3-3dab4dca5ae7-config-file\") pod \"limitador-limitador-84994f7ffc-w5fxt\" (UID: \"17604753-84a4-413e-aac3-3dab4dca5ae7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:03.022449 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:03.022411    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17604753-84a4-413e-aac3-3dab4dca5ae7-config-file\") pod \"limitador-limitador-84994f7ffc-w5fxt\" (UID: \"17604753-84a4-413e-aac3-3dab4dca5ae7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:03.022661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:03.022482    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vqk\" (UniqueName: \"kubernetes.io/projected/17604753-84a4-413e-aac3-3dab4dca5ae7-kube-api-access-55vqk\") pod \"limitador-limitador-84994f7ffc-w5fxt\" (UID: \"17604753-84a4-413e-aac3-3dab4dca5ae7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:03.023177 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:03.023150    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17604753-84a4-413e-aac3-3dab4dca5ae7-config-file\") pod \"limitador-limitador-84994f7ffc-w5fxt\" (UID: \"17604753-84a4-413e-aac3-3dab4dca5ae7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:03.032643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:03.032619    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vqk\" (UniqueName: \"kubernetes.io/projected/17604753-84a4-413e-aac3-3dab4dca5ae7-kube-api-access-55vqk\") pod \"limitador-limitador-84994f7ffc-w5fxt\" (UID: \"17604753-84a4-413e-aac3-3dab4dca5ae7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:03.157496 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:03.157443    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:03.434065 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:03.434038    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:40:03.435570 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:40:03.435536    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17604753_84a4_413e_aac3_3dab4dca5ae7.slice/crio-21e88d802494dd07bd263eb86ccd9aa9cd8eab8bffd2259e454643cf11348d34 WatchSource:0}: Error finding container 21e88d802494dd07bd263eb86ccd9aa9cd8eab8bffd2259e454643cf11348d34: Status 404 returned error can't find the container with id 21e88d802494dd07bd263eb86ccd9aa9cd8eab8bffd2259e454643cf11348d34
Sep 09 10:40:04.078058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:04.078019    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt" event={"ID":"17604753-84a4-413e-aac3-3dab4dca5ae7","Type":"ContainerStarted","Data":"21e88d802494dd07bd263eb86ccd9aa9cd8eab8bffd2259e454643cf11348d34"}
Sep 09 10:40:07.097633 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:07.097595    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt" event={"ID":"17604753-84a4-413e-aac3-3dab4dca5ae7","Type":"ContainerStarted","Data":"8083971cfb5f27794b7bc0b455bdbde2e843530a2bc456cc34146e224e1ce79b"}
Sep 09 10:40:07.098068 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:07.097677    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:07.114246 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:07.114192    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt" podStartSLOduration=2.506138679 podStartE2EDuration="5.114178235s" totalImagesPullingTime="2.608039556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:40:07.113586305 +0000 UTC m=+849.169145356" watchObservedRunningTime="2026-09-09 10:40:07.114178235 +0000 UTC m=+849.169737284"
Sep 09 10:40:18.104669 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:18.104632    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"
Sep 09 10:40:35.827220 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:35.827178    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7997c9c989-dnfq7"]
Sep 09 10:40:35.833681 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:35.833664    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7997c9c989-dnfq7"
Sep 09 10:40:35.835874 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:35.835848    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s8k29\""
Sep 09 10:40:35.837639 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:35.837611    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7997c9c989-dnfq7"]
Sep 09 10:40:35.868048 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:35.868022    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7997c9c989-dnfq7"]
Sep 09 10:40:35.868805 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:40:35.868773    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sld88], unattached volumes=[], failed to process volumes=[kube-api-access-sld88]: context canceled" pod="kuadrant-system/authorino-7997c9c989-dnfq7" podUID="90477fb1-a29e-4d2f-8091-21e2ca93b19b"
Sep 09 10:40:35.904528 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:35.904499    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sld88\" (UniqueName: \"kubernetes.io/projected/90477fb1-a29e-4d2f-8091-21e2ca93b19b-kube-api-access-sld88\") pod \"authorino-7997c9c989-dnfq7\" (UID: \"90477fb1-a29e-4d2f-8091-21e2ca93b19b\") " pod="kuadrant-system/authorino-7997c9c989-dnfq7"
Sep 09 10:40:36.005692 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:36.005650    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sld88\" (UniqueName: \"kubernetes.io/projected/90477fb1-a29e-4d2f-8091-21e2ca93b19b-kube-api-access-sld88\") pod \"authorino-7997c9c989-dnfq7\" (UID: \"90477fb1-a29e-4d2f-8091-21e2ca93b19b\") " pod="kuadrant-system/authorino-7997c9c989-dnfq7"
Sep 09 10:40:36.014156 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:36.014134    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sld88\" (UniqueName: \"kubernetes.io/projected/90477fb1-a29e-4d2f-8091-21e2ca93b19b-kube-api-access-sld88\") pod \"authorino-7997c9c989-dnfq7\" (UID: \"90477fb1-a29e-4d2f-8091-21e2ca93b19b\") " pod="kuadrant-system/authorino-7997c9c989-dnfq7"
Sep 09 10:40:36.258993 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:36.258960    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7997c9c989-dnfq7"
Sep 09 10:40:36.267244 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:36.267218    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7997c9c989-dnfq7"
Sep 09 10:40:36.308646 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:36.308609    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sld88\" (UniqueName: \"kubernetes.io/projected/90477fb1-a29e-4d2f-8091-21e2ca93b19b-kube-api-access-sld88\") pod \"90477fb1-a29e-4d2f-8091-21e2ca93b19b\" (UID: \"90477fb1-a29e-4d2f-8091-21e2ca93b19b\") "
Sep 09 10:40:36.310724 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:36.310688    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90477fb1-a29e-4d2f-8091-21e2ca93b19b-kube-api-access-sld88" (OuterVolumeSpecName: "kube-api-access-sld88") pod "90477fb1-a29e-4d2f-8091-21e2ca93b19b" (UID: "90477fb1-a29e-4d2f-8091-21e2ca93b19b"). InnerVolumeSpecName "kube-api-access-sld88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:40:36.409625 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:36.409591    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sld88\" (UniqueName: \"kubernetes.io/projected/90477fb1-a29e-4d2f-8091-21e2ca93b19b-kube-api-access-sld88\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:40:37.263646 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:37.263614    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7997c9c989-dnfq7"
Sep 09 10:40:37.289643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:37.289615    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7997c9c989-dnfq7"]
Sep 09 10:40:37.293306 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:37.293276    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7997c9c989-dnfq7"]
Sep 09 10:40:38.474211 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:40:38.474176    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90477fb1-a29e-4d2f-8091-21e2ca93b19b" path="/var/lib/kubelet/pods/90477fb1-a29e-4d2f-8091-21e2ca93b19b/volumes"
Sep 09 10:41:19.970113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:41:19.970071    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:41:45.500679 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:41:45.500643    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:42:09.007856 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:42:09.007820    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:42:32.048618 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:42:32.048530    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:42:47.805656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:42:47.805619    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:42:53.405676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:42:53.405642    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:43:02.518458 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:02.518425    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:43:14.706438 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.706402    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"]
Sep 09 10:43:14.711769 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.711736    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.715116 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.715092    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2mjpr\""
Sep 09 10:43:14.715591 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.715571    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 10:43:14.715709 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.715571    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 10:43:14.715709 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.715623    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 10:43:14.724856 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.724830    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"]
Sep 09 10:43:14.820796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.820769    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.820959 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.820832    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.820959 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.820869    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.820959 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.820908    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f32a40-dc73-461f-b2cb-52d88533eac0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.821070 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.820965    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k64fn\" (UniqueName: \"kubernetes.io/projected/f6f32a40-dc73-461f-b2cb-52d88533eac0-kube-api-access-k64fn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.821070 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.821013    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922114 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922079    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922122    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f32a40-dc73-461f-b2cb-52d88533eac0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922162    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k64fn\" (UniqueName: \"kubernetes.io/projected/f6f32a40-dc73-461f-b2cb-52d88533eac0-kube-api-access-k64fn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922199    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922232    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922476 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922325    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922531 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922485    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922665 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922644    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.922739 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.922670    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.924576 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.924551    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f32a40-dc73-461f-b2cb-52d88533eac0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.924665 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.924652    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f32a40-dc73-461f-b2cb-52d88533eac0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:14.930707 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:14.930684    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k64fn\" (UniqueName: \"kubernetes.io/projected/f6f32a40-dc73-461f-b2cb-52d88533eac0-kube-api-access-k64fn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2\" (UID: \"f6f32a40-dc73-461f-b2cb-52d88533eac0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:15.022907 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:15.022795    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:15.299246 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:15.299221    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"]
Sep 09 10:43:15.300930 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:43:15.300883    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6f32a40_dc73_461f_b2cb_52d88533eac0.slice/crio-c51717fa43c6adfddb61613895bf096310d5374b19c6ccf662f636115d77082b WatchSource:0}: Error finding container c51717fa43c6adfddb61613895bf096310d5374b19c6ccf662f636115d77082b: Status 404 returned error can't find the container with id c51717fa43c6adfddb61613895bf096310d5374b19c6ccf662f636115d77082b
Sep 09 10:43:15.305463 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:15.305440    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:43:16.139036 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:16.138992    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2" event={"ID":"f6f32a40-dc73-461f-b2cb-52d88533eac0","Type":"ContainerStarted","Data":"c51717fa43c6adfddb61613895bf096310d5374b19c6ccf662f636115d77082b"}
Sep 09 10:43:17.118547 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:17.118511    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:43:17.147081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:17.147044    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2" event={"ID":"f6f32a40-dc73-461f-b2cb-52d88533eac0","Type":"ContainerStarted","Data":"db015dc04113acf02ab6a2aee7f68619d7d4d63383c22e2e7c2484e9f83ede6e"}
Sep 09 10:43:17.170019 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:17.169963    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2" podStartSLOduration=2.170444804 podStartE2EDuration="3.169946716s" totalImagesPullingTime="999.501912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:43:17.168014071 +0000 UTC m=+1039.223573121" watchObservedRunningTime="2026-09-09 10:43:17.169946716 +0000 UTC m=+1039.225505766"
Sep 09 10:43:25.023154 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:25.023096    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:25.023154 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:25.023170    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:25.035751 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:25.035725    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:43:25.202317 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:43:25.202280    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2"
Sep 09 10:44:12.543920 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.543829    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-696d8d6478-k467s"]
Sep 09 10:44:12.547419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.547401    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-696d8d6478-k467s"
Sep 09 10:44:12.550334 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.550301    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s8k29\""
Sep 09 10:44:12.550441 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.550331    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 10:44:12.556239 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.556216    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-696d8d6478-k467s"]
Sep 09 10:44:12.662608 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.662571    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce43a5cb-a931-4444-a639-6a2ec9ce3257-tls-cert\") pod \"authorino-696d8d6478-k467s\" (UID: \"ce43a5cb-a931-4444-a639-6a2ec9ce3257\") " pod="kuadrant-system/authorino-696d8d6478-k467s"
Sep 09 10:44:12.662794 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.662618    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkbrt\" (UniqueName: \"kubernetes.io/projected/ce43a5cb-a931-4444-a639-6a2ec9ce3257-kube-api-access-wkbrt\") pod \"authorino-696d8d6478-k467s\" (UID: \"ce43a5cb-a931-4444-a639-6a2ec9ce3257\") " pod="kuadrant-system/authorino-696d8d6478-k467s"
Sep 09 10:44:12.763364 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.763321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce43a5cb-a931-4444-a639-6a2ec9ce3257-tls-cert\") pod \"authorino-696d8d6478-k467s\" (UID: \"ce43a5cb-a931-4444-a639-6a2ec9ce3257\") " pod="kuadrant-system/authorino-696d8d6478-k467s"
Sep 09 10:44:12.763364 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.763370    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkbrt\" (UniqueName: \"kubernetes.io/projected/ce43a5cb-a931-4444-a639-6a2ec9ce3257-kube-api-access-wkbrt\") pod \"authorino-696d8d6478-k467s\" (UID: \"ce43a5cb-a931-4444-a639-6a2ec9ce3257\") " pod="kuadrant-system/authorino-696d8d6478-k467s"
Sep 09 10:44:12.765871 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.765845    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce43a5cb-a931-4444-a639-6a2ec9ce3257-tls-cert\") pod \"authorino-696d8d6478-k467s\" (UID: \"ce43a5cb-a931-4444-a639-6a2ec9ce3257\") " pod="kuadrant-system/authorino-696d8d6478-k467s"
Sep 09 10:44:12.772471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.772449    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkbrt\" (UniqueName: \"kubernetes.io/projected/ce43a5cb-a931-4444-a639-6a2ec9ce3257-kube-api-access-wkbrt\") pod \"authorino-696d8d6478-k467s\" (UID: \"ce43a5cb-a931-4444-a639-6a2ec9ce3257\") " pod="kuadrant-system/authorino-696d8d6478-k467s"
Sep 09 10:44:12.858337 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:12.858226    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-696d8d6478-k467s"
Sep 09 10:44:13.128491 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:13.128464    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-696d8d6478-k467s"]
Sep 09 10:44:13.130326 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:44:13.130298    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce43a5cb_a931_4444_a639_6a2ec9ce3257.slice/crio-3e7f7bba8b6dc72bcb37125bef09c428b14420fcac2119613815ece507230331 WatchSource:0}: Error finding container 3e7f7bba8b6dc72bcb37125bef09c428b14420fcac2119613815ece507230331: Status 404 returned error can't find the container with id 3e7f7bba8b6dc72bcb37125bef09c428b14420fcac2119613815ece507230331
Sep 09 10:44:13.462021 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:13.461928    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-696d8d6478-k467s" event={"ID":"ce43a5cb-a931-4444-a639-6a2ec9ce3257","Type":"ContainerStarted","Data":"3e7f7bba8b6dc72bcb37125bef09c428b14420fcac2119613815ece507230331"}
Sep 09 10:44:20.506153 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:20.506114    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-696d8d6478-k467s" event={"ID":"ce43a5cb-a931-4444-a639-6a2ec9ce3257","Type":"ContainerStarted","Data":"16cb49bca6dd7c2bdc400752bddd3a4244a541d341958d3ab42cd217b920084c"}
Sep 09 10:44:20.525798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:20.525742    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-696d8d6478-k467s" podStartSLOduration=1.744659186 podStartE2EDuration="8.52572771s" totalImagesPullingTime="6.781068524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:20.522992724 +0000 UTC m=+1102.578551774" watchObservedRunningTime="2026-09-09 10:44:20.52572771 +0000 UTC m=+1102.581286759"
Sep 09 10:44:39.745176 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:39.745137    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:39.748747 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:39.748731    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:39.752041 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:39.752021    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dqfgn\""
Sep 09 10:44:39.758598 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:39.758575    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:39.808900 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:39.808868    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz66w\" (UniqueName: \"kubernetes.io/projected/7d9d1958-05c1-4114-b67b-679a9ef0df4a-kube-api-access-lz66w\") pod \"test-curl-7944-140487102024496\" (UID: \"7d9d1958-05c1-4114-b67b-679a9ef0df4a\") " pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:39.909725 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:39.909680    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz66w\" (UniqueName: \"kubernetes.io/projected/7d9d1958-05c1-4114-b67b-679a9ef0df4a-kube-api-access-lz66w\") pod \"test-curl-7944-140487102024496\" (UID: \"7d9d1958-05c1-4114-b67b-679a9ef0df4a\") " pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:39.917736 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:39.917712    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz66w\" (UniqueName: \"kubernetes.io/projected/7d9d1958-05c1-4114-b67b-679a9ef0df4a-kube-api-access-lz66w\") pod \"test-curl-7944-140487102024496\" (UID: \"7d9d1958-05c1-4114-b67b-679a9ef0df4a\") " pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:40.058359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:40.058278    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:40.327646 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:40.327620    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:40.329546 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:44:40.329516    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d9d1958_05c1_4114_b67b_679a9ef0df4a.slice/crio-9185a958af18ade6fd97b7637d4e80ee07742258e5aed56802b55477ebc475c2 WatchSource:0}: Error finding container 9185a958af18ade6fd97b7637d4e80ee07742258e5aed56802b55477ebc475c2: Status 404 returned error can't find the container with id 9185a958af18ade6fd97b7637d4e80ee07742258e5aed56802b55477ebc475c2
Sep 09 10:44:40.617688 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:40.617650    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7944-140487102024496" event={"ID":"7d9d1958-05c1-4114-b67b-679a9ef0df4a","Type":"ContainerStarted","Data":"9185a958af18ade6fd97b7637d4e80ee07742258e5aed56802b55477ebc475c2"}
Sep 09 10:44:41.423459 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:41.423425    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:44:43.641217 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:43.641179    2586 generic.go:356] "Generic (PLEG): container finished" podID="7d9d1958-05c1-4114-b67b-679a9ef0df4a" containerID="3d20621394bc85c445155962b5b90499d72f96de4e414c0794a3f7d0a12b6cd0" exitCode=0
Sep 09 10:44:43.641586 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:43.641269    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7944-140487102024496" event={"ID":"7d9d1958-05c1-4114-b67b-679a9ef0df4a","Type":"ContainerDied","Data":"3d20621394bc85c445155962b5b90499d72f96de4e414c0794a3f7d0a12b6cd0"}
Sep 09 10:44:44.777654 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.777626    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:44.825903 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.825867    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7944-140487102024496_7d9d1958-05c1-4114-b67b-679a9ef0df4a/test-curl-7944-140487102024496/0.log"
Sep 09 10:44:44.850946 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.850914    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:44.856757 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.856722    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:44.859425 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.859391    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz66w\" (UniqueName: \"kubernetes.io/projected/7d9d1958-05c1-4114-b67b-679a9ef0df4a-kube-api-access-lz66w\") pod \"7d9d1958-05c1-4114-b67b-679a9ef0df4a\" (UID: \"7d9d1958-05c1-4114-b67b-679a9ef0df4a\") "
Sep 09 10:44:44.862730 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.862696    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9d1958-05c1-4114-b67b-679a9ef0df4a-kube-api-access-lz66w" (OuterVolumeSpecName: "kube-api-access-lz66w") pod "7d9d1958-05c1-4114-b67b-679a9ef0df4a" (UID: "7d9d1958-05c1-4114-b67b-679a9ef0df4a"). InnerVolumeSpecName "kube-api-access-lz66w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:44:44.961236 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.961145    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz66w\" (UniqueName: \"kubernetes.io/projected/7d9d1958-05c1-4114-b67b-679a9ef0df4a-kube-api-access-lz66w\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:44:44.972739 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.972706    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:44.973170 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.973155    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9d1958-05c1-4114-b67b-679a9ef0df4a" containerName="test-curl-7944-140487102024496"
Sep 09 10:44:44.973219 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.973171    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9d1958-05c1-4114-b67b-679a9ef0df4a" containerName="test-curl-7944-140487102024496"
Sep 09 10:44:44.973255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.973241    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9d1958-05c1-4114-b67b-679a9ef0df4a" containerName="test-curl-7944-140487102024496"
Sep 09 10:44:44.976438 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.976421    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:44.984009 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:44.983874    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:45.062186 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.062154    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlzkw\" (UniqueName: \"kubernetes.io/projected/4c59cd30-d479-4e37-a809-bf9e7eeaab91-kube-api-access-jlzkw\") pod \"test-curl-7944-140487102024496\" (UID: \"4c59cd30-d479-4e37-a809-bf9e7eeaab91\") " pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:45.163658 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.163609    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzkw\" (UniqueName: \"kubernetes.io/projected/4c59cd30-d479-4e37-a809-bf9e7eeaab91-kube-api-access-jlzkw\") pod \"test-curl-7944-140487102024496\" (UID: \"4c59cd30-d479-4e37-a809-bf9e7eeaab91\") " pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:45.172664 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.172632    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzkw\" (UniqueName: \"kubernetes.io/projected/4c59cd30-d479-4e37-a809-bf9e7eeaab91-kube-api-access-jlzkw\") pod \"test-curl-7944-140487102024496\" (UID: \"4c59cd30-d479-4e37-a809-bf9e7eeaab91\") " pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:45.287365 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.287269    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:45.557858 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.557833    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:45.559480 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:44:45.559455    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c59cd30_d479_4e37_a809_bf9e7eeaab91.slice/crio-4371bc59ce602bcbe2c4298db39b73a7c447da84969c07e3be5627382a4b413c WatchSource:0}: Error finding container 4371bc59ce602bcbe2c4298db39b73a7c447da84969c07e3be5627382a4b413c: Status 404 returned error can't find the container with id 4371bc59ce602bcbe2c4298db39b73a7c447da84969c07e3be5627382a4b413c
Sep 09 10:44:45.653796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.653766    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9185a958af18ade6fd97b7637d4e80ee07742258e5aed56802b55477ebc475c2"
Sep 09 10:44:45.653796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.653783    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:45.655113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.655086    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7944-140487102024496" event={"ID":"4c59cd30-d479-4e37-a809-bf9e7eeaab91","Type":"ContainerStarted","Data":"4371bc59ce602bcbe2c4298db39b73a7c447da84969c07e3be5627382a4b413c"}
Sep 09 10:44:45.666251 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:45.666220    2586 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7944-140487102024496" oldPodUID="7d9d1958-05c1-4114-b67b-679a9ef0df4a" podUID="4c59cd30-d479-4e37-a809-bf9e7eeaab91"
Sep 09 10:44:46.475817 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:46.475741    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9d1958-05c1-4114-b67b-679a9ef0df4a" path="/var/lib/kubelet/pods/7d9d1958-05c1-4114-b67b-679a9ef0df4a/volumes"
Sep 09 10:44:46.661766 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:46.661734    2586 generic.go:356] "Generic (PLEG): container finished" podID="4c59cd30-d479-4e37-a809-bf9e7eeaab91" containerID="8ec85ca7865ca2ae1d308c7833b735cb7c19f383b9ec6fef2693f5f88e788f82" exitCode=0
Sep 09 10:44:46.661970 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:46.661835    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7944-140487102024496" event={"ID":"4c59cd30-d479-4e37-a809-bf9e7eeaab91","Type":"ContainerDied","Data":"8ec85ca7865ca2ae1d308c7833b735cb7c19f383b9ec6fef2693f5f88e788f82"}
Sep 09 10:44:47.800556 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:47.800530    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:47.824954 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:47.824923    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7944-140487102024496_4c59cd30-d479-4e37-a809-bf9e7eeaab91/test-curl-7944-140487102024496/0.log"
Sep 09 10:44:47.849150 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:47.849114    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:47.855724 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:47.855698    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7944-140487102024496"]
Sep 09 10:44:47.888813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:47.888778    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlzkw\" (UniqueName: \"kubernetes.io/projected/4c59cd30-d479-4e37-a809-bf9e7eeaab91-kube-api-access-jlzkw\") pod \"4c59cd30-d479-4e37-a809-bf9e7eeaab91\" (UID: \"4c59cd30-d479-4e37-a809-bf9e7eeaab91\") "
Sep 09 10:44:47.890827 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:47.890800    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c59cd30-d479-4e37-a809-bf9e7eeaab91-kube-api-access-jlzkw" (OuterVolumeSpecName: "kube-api-access-jlzkw") pod "4c59cd30-d479-4e37-a809-bf9e7eeaab91" (UID: "4c59cd30-d479-4e37-a809-bf9e7eeaab91"). InnerVolumeSpecName "kube-api-access-jlzkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:44:47.990286 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:47.990204    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlzkw\" (UniqueName: \"kubernetes.io/projected/4c59cd30-d479-4e37-a809-bf9e7eeaab91-kube-api-access-jlzkw\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:44:48.475249 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.475215    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c59cd30-d479-4e37-a809-bf9e7eeaab91" path="/var/lib/kubelet/pods/4c59cd30-d479-4e37-a809-bf9e7eeaab91/volumes"
Sep 09 10:44:48.676081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.676053    2586 scope.go:117] "RemoveContainer" containerID="8ec85ca7865ca2ae1d308c7833b735cb7c19f383b9ec6fef2693f5f88e788f82"
Sep 09 10:44:48.676081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.676065    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7944-140487102024496"
Sep 09 10:44:48.683484 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.683459    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"]
Sep 09 10:44:48.683983 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.683963    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c59cd30-d479-4e37-a809-bf9e7eeaab91" containerName="test-curl-7944-140487102024496"
Sep 09 10:44:48.683983 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.683983    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c59cd30-d479-4e37-a809-bf9e7eeaab91" containerName="test-curl-7944-140487102024496"
Sep 09 10:44:48.684158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.684054    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c59cd30-d479-4e37-a809-bf9e7eeaab91" containerName="test-curl-7944-140487102024496"
Sep 09 10:44:48.688522 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.688499    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.690682 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.690661    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4bbfe3-openshift-default-dockercfg-jgncj\""
Sep 09 10:44:48.703112 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.703092    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"]
Sep 09 10:44:48.796392 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-data\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.796392 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796365    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istiod-ca-cert\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.796566 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796398    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-envoy\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.796566 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796436    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sp9f\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-kube-api-access-5sp9f\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.796566 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796465    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-socket\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.796566 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-credential-socket\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.796693 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796635    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-certs\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.796693 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796671    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-podinfo\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.796693 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.796690    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-token\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897283 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-certs\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897304    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-podinfo\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897324    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-token\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897348    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-data\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897374    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istiod-ca-cert\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897392    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-envoy\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897413    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sp9f\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-kube-api-access-5sp9f\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-socket\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.897760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897617    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-credential-socket\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.898228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897779    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-certs\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.898228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.897879    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-socket\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.898228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.898040    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-data\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.898228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.898100    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-credential-socket\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.898228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.898207    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istiod-ca-cert\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.899817 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.899794    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-envoy\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.899974 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.899954    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-podinfo\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.906033 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.906004    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-token\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.906033 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.906023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sp9f\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-kube-api-access-5sp9f\") pod \"e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") " pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:48.998433 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:48.998397    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:49.282025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:49.281995    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"]
Sep 09 10:44:49.283712 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:44:49.283686    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ff2e9a_29e4_4cdc_b6b4_f2e782e5e79f.slice/crio-5f78717eda24a731962e0c5dcae8dcb94194bb1ed592d6d48bc85af2ea110fa6 WatchSource:0}: Error finding container 5f78717eda24a731962e0c5dcae8dcb94194bb1ed592d6d48bc85af2ea110fa6: Status 404 returned error can't find the container with id 5f78717eda24a731962e0c5dcae8dcb94194bb1ed592d6d48bc85af2ea110fa6
Sep 09 10:44:49.286656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:49.286617    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:49.286758 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:49.286707    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:49.286758 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:49.286745    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:44:49.687124 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:49.687090    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" event={"ID":"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f","Type":"ContainerStarted","Data":"fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884"}
Sep 09 10:44:49.687300 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:49.687151    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" event={"ID":"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f","Type":"ContainerStarted","Data":"5f78717eda24a731962e0c5dcae8dcb94194bb1ed592d6d48bc85af2ea110fa6"}
Sep 09 10:44:49.760815 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:49.760761    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" podStartSLOduration=1.760742818 podStartE2EDuration="1.760742818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:49.759116194 +0000 UTC m=+1131.814675256" watchObservedRunningTime="2026-09-09 10:44:49.760742818 +0000 UTC m=+1131.816301880"
Sep 09 10:44:49.999163 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:49.999079    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:50.615138 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.615103    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:44:50.926131 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.926045    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"]
Sep 09 10:44:50.930034 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.930011    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:50.936600 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.936576    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e52855bf-serving-cert\""
Sep 09 10:44:50.936755 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.936685    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 10:44:50.936906 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.936872    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k8bm4\""
Sep 09 10:44:50.937027 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.937012    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 10:44:50.937113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.937076    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 10:44:50.945042 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:50.945014    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"]
Sep 09 10:44:51.004073 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.004044    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:51.019067 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.019032    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-metrics-certs\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.019212 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.019114    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8lfm\" (UniqueName: \"kubernetes.io/projected/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-kube-api-access-d8lfm\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.019263 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.019208    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-openshift-service-ca\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.019263 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.019244    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.119958 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.119925    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-openshift-service-ca\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.120132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.119967    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.120132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.120012    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-metrics-certs\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.120132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.120089    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8lfm\" (UniqueName: \"kubernetes.io/projected/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-kube-api-access-d8lfm\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.120273 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:44:51.120124    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-serving-cert: secret "maas-api-e2e-ait-e52855bf-serving-cert" not found
Sep 09 10:44:51.120273 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:44:51.120210    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls podName:d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532 nodeName:}" failed. No retries permitted until 2026-09-09 10:44:51.620187295 +0000 UTC m=+1133.675746339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls") pod "maas-api-e2e-ait-e52855bf-68c6c99c97-84796" (UID: "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532") : secret "maas-api-e2e-ait-e52855bf-serving-cert" not found
Sep 09 10:44:51.120663 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.120642    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-openshift-service-ca\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.122472 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.122450    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-metrics-certs\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.135225 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.135190    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8lfm\" (UniqueName: \"kubernetes.io/projected/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-kube-api-access-d8lfm\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.625718 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.625678    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.628391 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.628287    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls\") pod \"maas-api-e2e-ait-e52855bf-68c6c99c97-84796\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:51.698766 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.698728    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:51.699803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.699782    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:44:51.843675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:51.843638    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:52.151141 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:52.150671    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"]
Sep 09 10:44:52.707058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:52.707016    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796" event={"ID":"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532","Type":"ContainerStarted","Data":"b655d288950a818b9fdf193286cf5b37468faa851c91f8c134f0c3ad6500ede8"}
Sep 09 10:44:54.722338 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:54.722305    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796" event={"ID":"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532","Type":"ContainerStarted","Data":"a5d1baeb4aa0d4dc1f85a344386ff0d56aa846f50c0744bcb674c59b73950e76"}
Sep 09 10:44:54.722705 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:54.722379    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:44:54.745539 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:44:54.745495    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796" podStartSLOduration=2.342788442 podStartE2EDuration="4.745479656s" totalImagesPullingTime="2.402691214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:54.744495729 +0000 UTC m=+1136.800054802" watchObservedRunningTime="2026-09-09 10:44:54.745479656 +0000 UTC m=+1136.801038706"
Sep 09 10:45:00.010397 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.010354    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"]
Sep 09 10:45:00.016278 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.016256    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.018807 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.018784    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-709c1971-kserve-self-signed-certs\""
Sep 09 10:45:00.035496 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.035458    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"]
Sep 09 10:45:00.172643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.172607    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"]
Sep 09 10:45:00.176750 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.176726    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"
Sep 09 10:45:00.179779 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.179759    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\""
Sep 09 10:45:00.186239 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.186221    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"
Sep 09 10:45:00.195540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.195516    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"]
Sep 09 10:45:00.200990 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.200283    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"]
Sep 09 10:45:00.200990 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.200427    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"
Sep 09 10:45:00.215012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.214617    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tmp-dir\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.215012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.214665    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tls-certs\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.215012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.214708    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-dshm\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.215012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.214766    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-model-cache\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.215012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.214813    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-home\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.215012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.214837    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brvl\" (UniqueName: \"kubernetes.io/projected/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-kube-api-access-9brvl\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.223644 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.223621    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"]
Sep 09 10:45:00.254917 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.245106    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"
Sep 09 10:45:00.315543 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.315408    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tmp-dir\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.315543 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.315441    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tls-certs\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.315543 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.315467    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-dshm\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.315543 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.315499    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-model-cache\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.315543 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.315528    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-home\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.315543 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.315545    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brvl\" (UniqueName: \"kubernetes.io/projected/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-kube-api-access-9brvl\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.316142 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.316122    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tmp-dir\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.316593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.316548    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-model-cache\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.316694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.316624    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-home\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.318609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.318554    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-dshm\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.319178 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.319152    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tls-certs\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.333061 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.328071    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brvl\" (UniqueName: \"kubernetes.io/projected/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-kube-api-access-9brvl\") pod \"e2e-conflict-a-709c1971-kserve-868b848598-khxlm\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") " pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.514317 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.514287    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"]
Sep 09 10:45:00.517016 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:45:00.516987    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66641e46_0b73_4fa4_a752_1f61b59f9180.slice/crio-a16acae9126f990d8c242d163e27c30b51624960086e802ab2962cffea30b31b WatchSource:0}: Error finding container a16acae9126f990d8c242d163e27c30b51624960086e802ab2962cffea30b31b: Status 404 returned error can't find the container with id a16acae9126f990d8c242d163e27c30b51624960086e802ab2962cffea30b31b
Sep 09 10:45:00.566225 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.566193    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"]
Sep 09 10:45:00.567112 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:45:00.567088    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14290167_912d_43f5_84cf_fc677d2fd768.slice/crio-137aede55a5ec441029072851f4ceb2bea0dc7a2f00e62362dc42aafdf9c9498 WatchSource:0}: Error finding container 137aede55a5ec441029072851f4ceb2bea0dc7a2f00e62362dc42aafdf9c9498: Status 404 returned error can't find the container with id 137aede55a5ec441029072851f4ceb2bea0dc7a2f00e62362dc42aafdf9c9498
Sep 09 10:45:00.626110 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.626076    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:00.753149 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.753119    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:45:00.763630 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.763595    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt" event={"ID":"14290167-912d-43f5-84cf-fc677d2fd768","Type":"ContainerStarted","Data":"137aede55a5ec441029072851f4ceb2bea0dc7a2f00e62362dc42aafdf9c9498"}
Sep 09 10:45:00.764811 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.764754    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29" event={"ID":"66641e46-0b73-4fa4-a752-1f61b59f9180","Type":"ContainerStarted","Data":"a16acae9126f990d8c242d163e27c30b51624960086e802ab2962cffea30b31b"}
Sep 09 10:45:00.916740 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:00.916710    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"]
Sep 09 10:45:00.917852 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:45:00.917818    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c6ff458_2c20_40d6_92c7_7fe348bf24a1.slice/crio-a1bd7f94bdd905cd578f263c2ebf8b19e80fb96b7ad0aa51219a14ba2c8df08d WatchSource:0}: Error finding container a1bd7f94bdd905cd578f263c2ebf8b19e80fb96b7ad0aa51219a14ba2c8df08d: Status 404 returned error can't find the container with id a1bd7f94bdd905cd578f263c2ebf8b19e80fb96b7ad0aa51219a14ba2c8df08d
Sep 09 10:45:01.773809 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:01.773773    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm" event={"ID":"8c6ff458-2c20-40d6-92c7-7fe348bf24a1","Type":"ContainerStarted","Data":"c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1"}
Sep 09 10:45:01.773809 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:01.773815    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm" event={"ID":"8c6ff458-2c20-40d6-92c7-7fe348bf24a1","Type":"ContainerStarted","Data":"a1bd7f94bdd905cd578f263c2ebf8b19e80fb96b7ad0aa51219a14ba2c8df08d"}
Sep 09 10:45:01.793876 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:01.793817    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm" podStartSLOduration=2.793800858 podStartE2EDuration="2.793800858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:01.790189433 +0000 UTC m=+1143.845748473" watchObservedRunningTime="2026-09-09 10:45:01.793800858 +0000 UTC m=+1143.849359901"
Sep 09 10:45:04.794852 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:04.794757    2586 generic.go:356] "Generic (PLEG): container finished" podID="14290167-912d-43f5-84cf-fc677d2fd768" containerID="6eebef8f7b0eeea39c94282e67cfff91bbfefaf78a8c92ef49f598f62808d7e5" exitCode=0
Sep 09 10:45:04.795278 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:04.794842    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt" event={"ID":"14290167-912d-43f5-84cf-fc677d2fd768","Type":"ContainerDied","Data":"6eebef8f7b0eeea39c94282e67cfff91bbfefaf78a8c92ef49f598f62808d7e5"}
Sep 09 10:45:04.796593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:04.796569    2586 generic.go:356] "Generic (PLEG): container finished" podID="66641e46-0b73-4fa4-a752-1f61b59f9180" containerID="38ecbb6c717c2afb2f61f1b1d965bf052ac0d257ff1b42c9bac317bf528a4327" exitCode=0
Sep 09 10:45:04.796722 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:04.796655    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29" event={"ID":"66641e46-0b73-4fa4-a752-1f61b59f9180","Type":"ContainerDied","Data":"38ecbb6c717c2afb2f61f1b1d965bf052ac0d257ff1b42c9bac317bf528a4327"}
Sep 09 10:45:05.946673 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:05.946641    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"
Sep 09 10:45:05.978407 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:05.978382    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"
Sep 09 10:45:06.750179 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.750090    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:45:06.766528 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.766482    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"]
Sep 09 10:45:06.767425 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.767398    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14290167-912d-43f5-84cf-fc677d2fd768" containerName="cleanup"
Sep 09 10:45:06.767584 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.767571    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="14290167-912d-43f5-84cf-fc677d2fd768" containerName="cleanup"
Sep 09 10:45:06.767711 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.767698    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66641e46-0b73-4fa4-a752-1f61b59f9180" containerName="cleanup"
Sep 09 10:45:06.767792 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.767782    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66641e46-0b73-4fa4-a752-1f61b59f9180" containerName="cleanup"
Sep 09 10:45:06.768024 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.768008    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="66641e46-0b73-4fa4-a752-1f61b59f9180" containerName="cleanup"
Sep 09 10:45:06.768205 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.768191    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="14290167-912d-43f5-84cf-fc677d2fd768" containerName="cleanup"
Sep 09 10:45:06.776839 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.776153    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.785920 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.782238    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-da3c64-openshift-default-dockercfg-58qrh\""
Sep 09 10:45:06.788826 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.788799    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"]
Sep 09 10:45:06.814013 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.813987    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"
Sep 09 10:45:06.814232 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.814206    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt" event={"ID":"14290167-912d-43f5-84cf-fc677d2fd768","Type":"ContainerDied","Data":"137aede55a5ec441029072851f4ceb2bea0dc7a2f00e62362dc42aafdf9c9498"}
Sep 09 10:45:06.814359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.814347    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="137aede55a5ec441029072851f4ceb2bea0dc7a2f00e62362dc42aafdf9c9498"
Sep 09 10:45:06.816249 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.816067    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29" event={"ID":"66641e46-0b73-4fa4-a752-1f61b59f9180","Type":"ContainerDied","Data":"a16acae9126f990d8c242d163e27c30b51624960086e802ab2962cffea30b31b"}
Sep 09 10:45:06.816249 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.816123    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a16acae9126f990d8c242d163e27c30b51624960086e802ab2962cffea30b31b"
Sep 09 10:45:06.816249 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.816182    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"
Sep 09 10:45:06.894384 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894340    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-data\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.894555 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894424    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-envoy\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.894555 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894464    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-certs\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.894555 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894494    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be21e8f-f131-42bd-934d-b3be0cf1c879-istiod-ca-cert\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.894555 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894546    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkx9g\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-kube-api-access-tkx9g\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.894714 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894619    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-podinfo\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.894714 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894705    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-credential-socket\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.894789 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894743    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-socket\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.894789 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.894774    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-token\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.995500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-podinfo\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995577    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-credential-socket\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995617    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-socket\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995641    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-token\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995750    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-data\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-envoy\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995834    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-certs\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995865    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be21e8f-f131-42bd-934d-b3be0cf1c879-istiod-ca-cert\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.995916    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkx9g\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-kube-api-access-tkx9g\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.996048    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-socket\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.996224    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-certs\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.996339    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-data\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.996614    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-credential-socket\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.996690    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be21e8f-f131-42bd-934d-b3be0cf1c879-istiod-ca-cert\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.998694    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-podinfo\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:06.999204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:06.998776    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-envoy\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:07.014190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.014123    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-token\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:07.015984 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.015956    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkx9g\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-kube-api-access-tkx9g\") pod \"e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") " pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:07.074208 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.074162    2586 patch_prober.go:28] interesting pod/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:48886->10.132.0.54:15021: read: connection reset by peer" start-of-body=
Sep 09 10:45:07.074348 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.074249    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:48886->10.132.0.54:15021: read: connection reset by peer"
Sep 09 10:45:07.104668 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.104640    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:07.403801 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.403755    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"]
Sep 09 10:45:07.404481 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:45:07.404447    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be21e8f_f131_42bd_934d_b3be0cf1c879.slice/crio-aa83f052b32270a161aa2cfe5f32f05840854ed9f6695a7daa84bb51b07ca4d3 WatchSource:0}: Error finding container aa83f052b32270a161aa2cfe5f32f05840854ed9f6695a7daa84bb51b07ca4d3: Status 404 returned error can't find the container with id aa83f052b32270a161aa2cfe5f32f05840854ed9f6695a7daa84bb51b07ca4d3
Sep 09 10:45:07.406904 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.406858    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.406984 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.406945    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.407024 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.406987    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.826160 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.826073    2586 generic.go:356] "Generic (PLEG): container finished" podID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerID="fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884" exitCode=0
Sep 09 10:45:07.826329 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.826149    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" event={"ID":"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f","Type":"ContainerDied","Data":"fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884"}
Sep 09 10:45:07.826726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.826702    2586 scope.go:117] "RemoveContainer" containerID="fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884"
Sep 09 10:45:07.828059 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.828012    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.828169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.828114    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.828169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.828141    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" event={"ID":"0be21e8f-f131-42bd-934d-b3be0cf1c879","Type":"ContainerStarted","Data":"11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff"}
Sep 09 10:45:07.828287 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.828170    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:07.828287 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:07.828190    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" event={"ID":"0be21e8f-f131-42bd-934d-b3be0cf1c879","Type":"ContainerStarted","Data":"aa83f052b32270a161aa2cfe5f32f05840854ed9f6695a7daa84bb51b07ca4d3"}
Sep 09 10:45:08.104863 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:08.104775    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:08.106330 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:08.106301    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 10:45:08.106476 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:08.106349    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 10:45:08.840988 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:08.840951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" event={"ID":"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f","Type":"ContainerStarted","Data":"6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d"}
Sep 09 10:45:08.866227 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:08.866169    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podStartSLOduration=2.866151343 podStartE2EDuration="2.866151343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:07.877752192 +0000 UTC m=+1149.933311244" watchObservedRunningTime="2026-09-09 10:45:08.866151343 +0000 UTC m=+1150.921710394"
Sep 09 10:45:08.999264 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:08.999225    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:45:08.999494 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:08.999470    2586 patch_prober.go:28] interesting pod/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:45:08.999550 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:08.999521    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 10:45:09.109015 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:09.108938    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:09.850481 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:09.850445    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:09.854604 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:09.854577    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:09.999026 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:09.998991    2586 patch_prober.go:28] interesting pod/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:45:09.999190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:09.999045    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 10:45:10.627004 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:10.626970    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:10.627369 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:10.627023    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:10.662940 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:10.662880    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:10.869878 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:10.869840    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:45:10.999148 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:10.999060    2586 patch_prober.go:28] interesting pod/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:45:10.999148 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:10.999130    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 10:45:11.144915 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.139920    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"]
Sep 09 10:45:11.156761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.156084    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"]
Sep 09 10:45:11.156761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.156226    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.162228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.162202    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-da3c64-serving-cert\""
Sep 09 10:45:11.258976 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.258427    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-metrics-certs\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.258976 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.258490    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.258976 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.258543    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a758eed-a041-49b2-84f4-54b2803f495f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.258976 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.258656    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlgl8\" (UniqueName: \"kubernetes.io/projected/7a758eed-a041-49b2-84f4-54b2803f495f-kube-api-access-tlgl8\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.360140 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.360095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-metrics-certs\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.360331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.360151    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.360331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.360201    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a758eed-a041-49b2-84f4-54b2803f495f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.360331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.360295    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlgl8\" (UniqueName: \"kubernetes.io/projected/7a758eed-a041-49b2-84f4-54b2803f495f-kube-api-access-tlgl8\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.360788 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:45:11.360762    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-serving-cert: secret "maas-api-e2e-inf-b-da3c64-serving-cert" not found
Sep 09 10:45:11.361081 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:45:11.361066    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls podName:7a758eed-a041-49b2-84f4-54b2803f495f nodeName:}" failed. No retries permitted until 2026-09-09 10:45:11.861042832 +0000 UTC m=+1153.916601864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls") pod "maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb" (UID: "7a758eed-a041-49b2-84f4-54b2803f495f") : secret "maas-api-e2e-inf-b-da3c64-serving-cert" not found
Sep 09 10:45:11.361288 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.361267    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a758eed-a041-49b2-84f4-54b2803f495f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.362843 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.362822    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-metrics-certs\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.369798 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.369773    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlgl8\" (UniqueName: \"kubernetes.io/projected/7a758eed-a041-49b2-84f4-54b2803f495f-kube-api-access-tlgl8\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.866447 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.866409    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.868728 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.868708    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls\") pod \"maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:11.999776 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.999739    2586 patch_prober.go:28] interesting pod/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:45:11.999959 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:11.999811    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 10:45:12.072215 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:12.072173    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:12.350933 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:12.350524    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"]
Sep 09 10:45:12.353916 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:45:12.352062    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a758eed_a041_49b2_84f4_54b2803f495f.slice/crio-b5404891de4199a2b4e6c2a3db447b5e3b4a3c1af77aed44bc5cb046466c82fa WatchSource:0}: Error finding container b5404891de4199a2b4e6c2a3db447b5e3b4a3c1af77aed44bc5cb046466c82fa: Status 404 returned error can't find the container with id b5404891de4199a2b4e6c2a3db447b5e3b4a3c1af77aed44bc5cb046466c82fa
Sep 09 10:45:12.870109 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:12.870071    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb" event={"ID":"7a758eed-a041-49b2-84f4-54b2803f495f","Type":"ContainerStarted","Data":"b5404891de4199a2b4e6c2a3db447b5e3b4a3c1af77aed44bc5cb046466c82fa"}
Sep 09 10:45:12.999267 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:12.999235    2586 patch_prober.go:28] interesting pod/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 10:45:12.999415 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:12.999299    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 10:45:13.877944 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:13.877907    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb" event={"ID":"7a758eed-a041-49b2-84f4-54b2803f495f","Type":"ContainerStarted","Data":"38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b"}
Sep 09 10:45:13.878354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:13.878030    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:13.898345 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:13.898292    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb" podStartSLOduration=2.374324723 podStartE2EDuration="2.898270813s" totalImagesPullingTime="523.94609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:13.896411477 +0000 UTC m=+1155.951970528" watchObservedRunningTime="2026-09-09 10:45:13.898270813 +0000 UTC m=+1155.953829861"
Sep 09 10:45:14.002435 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:14.002408    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:45:14.002803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:14.002784    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:45:14.003489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:14.003468    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:45:19.889799 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:19.889769    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:45:21.805099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:21.805010    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"]
Sep 09 10:45:21.805529 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:21.805267    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796" podUID="d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" containerName="maas-api" containerID="cri-o://a5d1baeb4aa0d4dc1f85a344386ff0d56aa846f50c0744bcb674c59b73950e76" gracePeriod=30
Sep 09 10:45:21.940876 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:21.940580    2586 generic.go:356] "Generic (PLEG): container finished" podID="d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" containerID="a5d1baeb4aa0d4dc1f85a344386ff0d56aa846f50c0744bcb674c59b73950e76" exitCode=0
Sep 09 10:45:21.940876 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:21.940740    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796" event={"ID":"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532","Type":"ContainerDied","Data":"a5d1baeb4aa0d4dc1f85a344386ff0d56aa846f50c0744bcb674c59b73950e76"}
Sep 09 10:45:22.092792 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.092767    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:45:22.177624 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.177588    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls\") pod \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") "
Sep 09 10:45:22.177768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.177655    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-openshift-service-ca\") pod \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") "
Sep 09 10:45:22.177768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.177737    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-metrics-certs\") pod \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") "
Sep 09 10:45:22.177768 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.177765    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8lfm\" (UniqueName: \"kubernetes.io/projected/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-kube-api-access-d8lfm\") pod \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\" (UID: \"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532\") "
Sep 09 10:45:22.178045 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.178019    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" (UID: "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:45:22.179953 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.179925    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" (UID: "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:45:22.179953 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.179942    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-kube-api-access-d8lfm" (OuterVolumeSpecName: "kube-api-access-d8lfm") pod "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" (UID: "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532"). InnerVolumeSpecName "kube-api-access-d8lfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:45:22.180105 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.179948    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" (UID: "d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:45:22.278528 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.278483    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:45:22.278528 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.278516    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8lfm\" (UniqueName: \"kubernetes.io/projected/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-kube-api-access-d8lfm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:45:22.278528 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.278532    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:45:22.278839 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.278544    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:45:22.957291 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.957256    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796" event={"ID":"d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532","Type":"ContainerDied","Data":"b655d288950a818b9fdf193286cf5b37468faa851c91f8c134f0c3ad6500ede8"}
Sep 09 10:45:22.957710 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.957331    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"
Sep 09 10:45:22.957710 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.957344    2586 scope.go:117] "RemoveContainer" containerID="a5d1baeb4aa0d4dc1f85a344386ff0d56aa846f50c0744bcb674c59b73950e76"
Sep 09 10:45:22.976460 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.976437    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"]
Sep 09 10:45:22.982943 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:22.982920    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e52855bf-68c6c99c97-84796"]
Sep 09 10:45:24.475731 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:24.475693    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" path="/var/lib/kubelet/pods/d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532/volumes"
Sep 09 10:45:25.892089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:25.892037    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 10:45:25.892619 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:25.892112    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 10:45:26.130922 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:26.130877    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:45:27.993375 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:27.993346    2586 generic.go:356] "Generic (PLEG): container finished" podID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerID="11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff" exitCode=0
Sep 09 10:45:27.993796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:27.993425    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" event={"ID":"0be21e8f-f131-42bd-934d-b3be0cf1c879","Type":"ContainerDied","Data":"11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff"}
Sep 09 10:45:27.993878 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:27.993862    2586 scope.go:117] "RemoveContainer" containerID="11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff"
Sep 09 10:45:27.995035 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:27.995009    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:27.995118 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:27.995079    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:27.995155 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:27.995118    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:45:29.009281 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:29.009243    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" event={"ID":"0be21e8f-f131-42bd-934d-b3be0cf1c879","Type":"ContainerStarted","Data":"70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c"}
Sep 09 10:45:29.105092 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:29.105054    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:29.105333 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:29.105310    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 10:45:29.105382 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:29.105360    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 10:45:30.105760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:30.105721    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 10:45:30.106242 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:30.105782    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 10:45:31.105616 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:31.105576    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 10:45:31.105831 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:31.105642    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 10:45:32.105629 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:32.105591    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 10:45:32.105811 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:32.105655    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 10:45:33.106006 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:33.105966    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 10:45:33.106393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:33.106031    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 10:45:34.105766 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:34.105728    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 10:45:34.105972 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:34.105786    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 10:45:35.109152 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:35.109117    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:35.109517 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:35.109495    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:35.110121 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:35.110101    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:45:42.599818 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.599781    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"]
Sep 09 10:45:42.600266 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.600251    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" containerName="maas-api"
Sep 09 10:45:42.600324 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.600267    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" containerName="maas-api"
Sep 09 10:45:42.600369 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.600354    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6aa4dc3-7e8d-4e6d-b4e0-f280f3ef0532" containerName="maas-api"
Sep 09 10:45:42.603592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.603576    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.605672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.605649    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-709c1971-kserve-self-signed-certs\""
Sep 09 10:45:42.612965 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.612942    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"]
Sep 09 10:45:42.683371 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.683341    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-dshm\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.683371 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.683371    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-model-cache\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.683569 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.683402    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-home\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.683569 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.683493    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tmp-dir\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.683569 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.683543    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgvp\" (UniqueName: \"kubernetes.io/projected/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-kube-api-access-gjgvp\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.683667 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.683599    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tls-certs\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784208 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-dshm\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784208 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784206    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-model-cache\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784237    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-home\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784270    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tmp-dir\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784294    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgvp\" (UniqueName: \"kubernetes.io/projected/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-kube-api-access-gjgvp\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784464 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784324    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tls-certs\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784727 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784700    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-home\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784795 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784720    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tmp-dir\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.784795 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.784748    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-model-cache\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.786491 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.786462    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-dshm\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.786774 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.786755    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tls-certs\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.793482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.793451    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgvp\" (UniqueName: \"kubernetes.io/projected/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-kube-api-access-gjgvp\") pod \"e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") " pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:42.914352 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:42.914317    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:43.188355 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:43.188324    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"]
Sep 09 10:45:43.189407 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:45:43.189378    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e4ac93a_2e49_48cc_8597_4df2c4487ac9.slice/crio-9e56d759b77651cee1a5c034f6f6ec6538d243ebed71bc92a62661cf95795fd3 WatchSource:0}: Error finding container 9e56d759b77651cee1a5c034f6f6ec6538d243ebed71bc92a62661cf95795fd3: Status 404 returned error can't find the container with id 9e56d759b77651cee1a5c034f6f6ec6538d243ebed71bc92a62661cf95795fd3
Sep 09 10:45:43.698865 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:43.698825    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:45:44.113026 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:44.112988    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn" event={"ID":"1e4ac93a-2e49-48cc-8597-4df2c4487ac9","Type":"ContainerStarted","Data":"9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc"}
Sep 09 10:45:44.113200 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:44.113043    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn" event={"ID":"1e4ac93a-2e49-48cc-8597-4df2c4487ac9","Type":"ContainerStarted","Data":"9e56d759b77651cee1a5c034f6f6ec6538d243ebed71bc92a62661cf95795fd3"}
Sep 09 10:45:44.133625 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:44.133565    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn" podStartSLOduration=2.133548235 podStartE2EDuration="2.133548235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:44.131726592 +0000 UTC m=+1186.187285644" watchObservedRunningTime="2026-09-09 10:45:44.133548235 +0000 UTC m=+1186.189107285"
Sep 09 10:45:52.914656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:52.914602    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:52.914656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:52.914668    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:52.927522 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:52.927498    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:53.177444 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:53.177356    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:45:56.404498 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:45:56.404464    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:46:14.718652 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:14.718615    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:46:26.746931 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:26.746879    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"]
Sep 09 10:46:26.752079 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:26.752057    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"
Sep 09 10:46:26.754357 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:26.754339    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\""
Sep 09 10:46:26.759482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:26.759455    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"]
Sep 09 10:46:26.908482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:26.908428    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc802a-30cf-44a2-8a74-4302d7479427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k\" (UID: \"87cc802a-30cf-44a2-8a74-4302d7479427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"
Sep 09 10:46:27.010007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:27.009874    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc802a-30cf-44a2-8a74-4302d7479427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k\" (UID: \"87cc802a-30cf-44a2-8a74-4302d7479427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"
Sep 09 10:46:27.010552 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:27.010533    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc802a-30cf-44a2-8a74-4302d7479427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k\" (UID: \"87cc802a-30cf-44a2-8a74-4302d7479427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"
Sep 09 10:46:27.064373 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:27.064339    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"
Sep 09 10:46:27.339999 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:27.339974    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"]
Sep 09 10:46:27.342091 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:46:27.342064    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cc802a_30cf_44a2_8a74_4302d7479427.slice/crio-8181bf56818fae71cea946e55c7ac8af44b678d3a6a28e02705026528b37789d WatchSource:0}: Error finding container 8181bf56818fae71cea946e55c7ac8af44b678d3a6a28e02705026528b37789d: Status 404 returned error can't find the container with id 8181bf56818fae71cea946e55c7ac8af44b678d3a6a28e02705026528b37789d
Sep 09 10:46:27.371423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:27.371392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k" event={"ID":"87cc802a-30cf-44a2-8a74-4302d7479427","Type":"ContainerStarted","Data":"8181bf56818fae71cea946e55c7ac8af44b678d3a6a28e02705026528b37789d"}
Sep 09 10:46:28.379327 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:28.379238    2586 generic.go:356] "Generic (PLEG): container finished" podID="87cc802a-30cf-44a2-8a74-4302d7479427" containerID="00ca41c3415113213fb0aca6f40eb580e96c58d83bd83f9b268b89e1472f19a6" exitCode=0
Sep 09 10:46:28.379327 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:28.379315    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k" event={"ID":"87cc802a-30cf-44a2-8a74-4302d7479427","Type":"ContainerDied","Data":"00ca41c3415113213fb0aca6f40eb580e96c58d83bd83f9b268b89e1472f19a6"}
Sep 09 10:46:29.516749 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:29.516726    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"
Sep 09 10:46:29.634542 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:29.634509    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc802a-30cf-44a2-8a74-4302d7479427-maas-api-service-ca\") pod \"87cc802a-30cf-44a2-8a74-4302d7479427\" (UID: \"87cc802a-30cf-44a2-8a74-4302d7479427\") "
Sep 09 10:46:29.634923 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:29.634866    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cc802a-30cf-44a2-8a74-4302d7479427-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87cc802a-30cf-44a2-8a74-4302d7479427" (UID: "87cc802a-30cf-44a2-8a74-4302d7479427"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:46:29.735446 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:29.735363    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc802a-30cf-44a2-8a74-4302d7479427-maas-api-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:30.393478 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:30.393452    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"
Sep 09 10:46:30.393642 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:30.393451    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k" event={"ID":"87cc802a-30cf-44a2-8a74-4302d7479427","Type":"ContainerDied","Data":"8181bf56818fae71cea946e55c7ac8af44b678d3a6a28e02705026528b37789d"}
Sep 09 10:46:30.393642 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:30.393580    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8181bf56818fae71cea946e55c7ac8af44b678d3a6a28e02705026528b37789d"
Sep 09 10:46:30.612024 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:30.611985    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"]
Sep 09 10:46:33.355696 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:33.355652    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"]
Sep 09 10:46:33.359368 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:33.359345    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dfb9af4b-wpm9k"]
Sep 09 10:46:34.475776 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:34.475733    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87cc802a-30cf-44a2-8a74-4302d7479427" path="/var/lib/kubelet/pods/87cc802a-30cf-44a2-8a74-4302d7479427/volumes"
Sep 09 10:46:35.029260 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:35.029222    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"]
Sep 09 10:46:35.031847 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:35.031823    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4bbfe3-29815845-6mj29"]
Sep 09 10:46:35.223193 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:35.223157    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:46:35.434471 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:35.434436    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"]
Sep 09 10:46:36.474980 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:36.474948    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66641e46-0b73-4fa4-a752-1f61b59f9180" path="/var/lib/kubelet/pods/66641e46-0b73-4fa4-a752-1f61b59f9180/volumes"
Sep 09 10:46:45.632102 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.632051    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn" podUID="1e4ac93a-2e49-48cc-8597-4df2c4487ac9" containerName="main" containerID="cri-o://9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc" gracePeriod=45
Sep 09 10:46:45.879794 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.879769    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:46:45.985672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.985585    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-dshm\") pod \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") "
Sep 09 10:46:45.985672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.985624    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tls-certs\") pod \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") "
Sep 09 10:46:45.985672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.985660    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tmp-dir\") pod \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") "
Sep 09 10:46:45.985991 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.985707    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-model-cache\") pod \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") "
Sep 09 10:46:45.985991 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.985753    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjgvp\" (UniqueName: \"kubernetes.io/projected/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-kube-api-access-gjgvp\") pod \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") "
Sep 09 10:46:45.985991 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.985807    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-home\") pod \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\" (UID: \"1e4ac93a-2e49-48cc-8597-4df2c4487ac9\") "
Sep 09 10:46:45.986136 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.986008    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e4ac93a-2e49-48cc-8597-4df2c4487ac9" (UID: "1e4ac93a-2e49-48cc-8597-4df2c4487ac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:45.986136 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.986035    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e4ac93a-2e49-48cc-8597-4df2c4487ac9" (UID: "1e4ac93a-2e49-48cc-8597-4df2c4487ac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:45.986254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.986206    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-home" (OuterVolumeSpecName: "home") pod "1e4ac93a-2e49-48cc-8597-4df2c4487ac9" (UID: "1e4ac93a-2e49-48cc-8597-4df2c4487ac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:45.986254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.986235    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:45.986363 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.986257    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:45.988002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.987970    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-kube-api-access-gjgvp" (OuterVolumeSpecName: "kube-api-access-gjgvp") pod "1e4ac93a-2e49-48cc-8597-4df2c4487ac9" (UID: "1e4ac93a-2e49-48cc-8597-4df2c4487ac9"). InnerVolumeSpecName "kube-api-access-gjgvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:45.988002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.987977    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-dshm" (OuterVolumeSpecName: "dshm") pod "1e4ac93a-2e49-48cc-8597-4df2c4487ac9" (UID: "1e4ac93a-2e49-48cc-8597-4df2c4487ac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:45.988152 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:45.987980    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e4ac93a-2e49-48cc-8597-4df2c4487ac9" (UID: "1e4ac93a-2e49-48cc-8597-4df2c4487ac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:46:46.087600 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.087564    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjgvp\" (UniqueName: \"kubernetes.io/projected/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-kube-api-access-gjgvp\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:46.087600 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.087593    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:46.087600 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.087605    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:46.087843 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.087612    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4ac93a-2e49-48cc-8597-4df2c4487ac9-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:46.504026 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.503994    2586 generic.go:356] "Generic (PLEG): container finished" podID="1e4ac93a-2e49-48cc-8597-4df2c4487ac9" containerID="9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc" exitCode=0
Sep 09 10:46:46.504180 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.504039    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"
Sep 09 10:46:46.504180 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.504042    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn" event={"ID":"1e4ac93a-2e49-48cc-8597-4df2c4487ac9","Type":"ContainerDied","Data":"9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc"}
Sep 09 10:46:46.504180 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.504170    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn" event={"ID":"1e4ac93a-2e49-48cc-8597-4df2c4487ac9","Type":"ContainerDied","Data":"9e56d759b77651cee1a5c034f6f6ec6538d243ebed71bc92a62661cf95795fd3"}
Sep 09 10:46:46.504366 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.504204    2586 scope.go:117] "RemoveContainer" containerID="9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc"
Sep 09 10:46:46.513935 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.513917    2586 scope.go:117] "RemoveContainer" containerID="9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc"
Sep 09 10:46:46.514221 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:46:46.514199    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc\": container with ID starting with 9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc not found: ID does not exist" containerID="9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc"
Sep 09 10:46:46.514290 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.514233    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc"} err="failed to get container status \"9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc\": rpc error: code = NotFound desc = could not find container \"9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc\": container with ID starting with 9d16448f7ac8157b0eed0b8c2336bffaff1fad51ed8453d3568a905cd7960adc not found: ID does not exist"
Sep 09 10:46:46.522134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.522113    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"]
Sep 09 10:46:46.526792 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:46.526768    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-709c1971-kserve-6bbbbf56c6-ggtzn"]
Sep 09 10:46:48.476018 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:48.475979    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e4ac93a-2e49-48cc-8597-4df2c4487ac9" path="/var/lib/kubelet/pods/1e4ac93a-2e49-48cc-8597-4df2c4487ac9/volumes"
Sep 09 10:46:50.455232 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.455193    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm" podUID="8c6ff458-2c20-40d6-92c7-7fe348bf24a1" containerName="main" containerID="cri-o://c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1" gracePeriod=45
Sep 09 10:46:50.715012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.714955    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:46:50.834732 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.834704    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9brvl\" (UniqueName: \"kubernetes.io/projected/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-kube-api-access-9brvl\") pod \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") "
Sep 09 10:46:50.834732 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.834735    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-dshm\") pod \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") "
Sep 09 10:46:50.834947 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.834776    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tls-certs\") pod \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") "
Sep 09 10:46:50.834992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.834963    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-home\") pod \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") "
Sep 09 10:46:50.835099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.835083    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-model-cache\") pod \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") "
Sep 09 10:46:50.835161 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.835119    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tmp-dir\") pod \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\" (UID: \"8c6ff458-2c20-40d6-92c7-7fe348bf24a1\") "
Sep 09 10:46:50.835261 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.835236    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-home" (OuterVolumeSpecName: "home") pod "8c6ff458-2c20-40d6-92c7-7fe348bf24a1" (UID: "8c6ff458-2c20-40d6-92c7-7fe348bf24a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:50.835400 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.835377    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c6ff458-2c20-40d6-92c7-7fe348bf24a1" (UID: "8c6ff458-2c20-40d6-92c7-7fe348bf24a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:50.835467 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.835414    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c6ff458-2c20-40d6-92c7-7fe348bf24a1" (UID: "8c6ff458-2c20-40d6-92c7-7fe348bf24a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:50.835534 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.835520    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:50.835595 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.835553    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:50.835595 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.835561    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:50.836962 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.836933    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-kube-api-access-9brvl" (OuterVolumeSpecName: "kube-api-access-9brvl") pod "8c6ff458-2c20-40d6-92c7-7fe348bf24a1" (UID: "8c6ff458-2c20-40d6-92c7-7fe348bf24a1"). InnerVolumeSpecName "kube-api-access-9brvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:46:50.836962 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.836951    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c6ff458-2c20-40d6-92c7-7fe348bf24a1" (UID: "8c6ff458-2c20-40d6-92c7-7fe348bf24a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:46:50.837156 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.836944    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-dshm" (OuterVolumeSpecName: "dshm") pod "8c6ff458-2c20-40d6-92c7-7fe348bf24a1" (UID: "8c6ff458-2c20-40d6-92c7-7fe348bf24a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:46:50.937053 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.937004    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brvl\" (UniqueName: \"kubernetes.io/projected/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-kube-api-access-9brvl\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:50.937053 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.937048    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:50.937053 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:50.937058    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6ff458-2c20-40d6-92c7-7fe348bf24a1-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:46:51.537191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.537154    2586 generic.go:356] "Generic (PLEG): container finished" podID="8c6ff458-2c20-40d6-92c7-7fe348bf24a1" containerID="c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1" exitCode=0
Sep 09 10:46:51.537191 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.537195    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"
Sep 09 10:46:51.537661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.537217    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm" event={"ID":"8c6ff458-2c20-40d6-92c7-7fe348bf24a1","Type":"ContainerDied","Data":"c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1"}
Sep 09 10:46:51.537661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.537253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm" event={"ID":"8c6ff458-2c20-40d6-92c7-7fe348bf24a1","Type":"ContainerDied","Data":"a1bd7f94bdd905cd578f263c2ebf8b19e80fb96b7ad0aa51219a14ba2c8df08d"}
Sep 09 10:46:51.537661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.537275    2586 scope.go:117] "RemoveContainer" containerID="c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1"
Sep 09 10:46:51.547753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.547734    2586 scope.go:117] "RemoveContainer" containerID="c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1"
Sep 09 10:46:51.548069 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:46:51.548045    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1\": container with ID starting with c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1 not found: ID does not exist" containerID="c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1"
Sep 09 10:46:51.548175 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.548077    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1"} err="failed to get container status \"c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1\": rpc error: code = NotFound desc = could not find container \"c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1\": container with ID starting with c5e5244b7d1c07654712a5eeaf36c2d24efafb601e62bf2c016f49608079f6c1 not found: ID does not exist"
Sep 09 10:46:51.558005 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.557984    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"]
Sep 09 10:46:51.565436 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:51.565410    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-709c1971-kserve-868b848598-khxlm"]
Sep 09 10:46:52.475125 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:52.475094    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c6ff458-2c20-40d6-92c7-7fe348bf24a1" path="/var/lib/kubelet/pods/8c6ff458-2c20-40d6-92c7-7fe348bf24a1/volumes"
Sep 09 10:46:54.415082 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:46:54.415047    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:47:02.606376 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606341    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"]
Sep 09 10:47:02.606867 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606828    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6ff458-2c20-40d6-92c7-7fe348bf24a1" containerName="main"
Sep 09 10:47:02.606867 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606843    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6ff458-2c20-40d6-92c7-7fe348bf24a1" containerName="main"
Sep 09 10:47:02.606867 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606857    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4ac93a-2e49-48cc-8597-4df2c4487ac9" containerName="main"
Sep 09 10:47:02.606867 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606863    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4ac93a-2e49-48cc-8597-4df2c4487ac9" containerName="main"
Sep 09 10:47:02.607113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606877    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87cc802a-30cf-44a2-8a74-4302d7479427" containerName="revoke-keys"
Sep 09 10:47:02.607113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606883    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="87cc802a-30cf-44a2-8a74-4302d7479427" containerName="revoke-keys"
Sep 09 10:47:02.607113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606982    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="87cc802a-30cf-44a2-8a74-4302d7479427" containerName="revoke-keys"
Sep 09 10:47:02.607113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.606994    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4ac93a-2e49-48cc-8597-4df2c4487ac9" containerName="main"
Sep 09 10:47:02.607113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.607003    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6ff458-2c20-40d6-92c7-7fe348bf24a1" containerName="main"
Sep 09 10:47:02.611577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.611554    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.613812 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.613792    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-da3c64\"/\"kube-root-ca.crt\""
Sep 09 10:47:02.613812 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.613798    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-da3c64\"/\"openshift-service-ca.crt\""
Sep 09 10:47:02.614081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.613878    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da3c64\"/\"test-model-da3c64-kserve-self-signed-certs\""
Sep 09 10:47:02.614081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.613928    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da3c64\"/\"default-dockercfg-mjk4v\""
Sep 09 10:47:02.619094 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.618867    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"]
Sep 09 10:47:02.753087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.753054    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-dshm\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.753262 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.753100    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tmp-dir\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.753262 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.753166    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tls-certs\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.753262 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.753220    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-home\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.753368 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.753286    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkzhj\" (UniqueName: \"kubernetes.io/projected/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-kube-api-access-hkzhj\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.753368 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.753306    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-model-cache\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854130    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkzhj\" (UniqueName: \"kubernetes.io/projected/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-kube-api-access-hkzhj\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854170    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-model-cache\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-dshm\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854232    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tmp-dir\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854261    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tls-certs\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854289    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-home\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854655 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854630    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-model-cache\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854671    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tmp-dir\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.854719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.854707    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-home\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.856551 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.856503    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-dshm\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.856784 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.856767    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tls-certs\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.868110 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.868086    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkzhj\" (UniqueName: \"kubernetes.io/projected/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-kube-api-access-hkzhj\") pod \"test-model-da3c64-kserve-7f878b7779-4jhbp\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") " pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:02.922194 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:02.922152    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:03.199293 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:03.199268    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"]
Sep 09 10:47:03.200525 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:47:03.200501    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a1c9515_bdee_4eb3_9198_39fcf3283ab3.slice/crio-7b74c999b91b1bfdb5a17b85915da3a43c2766347bf1cbd1e0979766eba590d4 WatchSource:0}: Error finding container 7b74c999b91b1bfdb5a17b85915da3a43c2766347bf1cbd1e0979766eba590d4: Status 404 returned error can't find the container with id 7b74c999b91b1bfdb5a17b85915da3a43c2766347bf1cbd1e0979766eba590d4
Sep 09 10:47:03.616033 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:03.616000    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp" event={"ID":"0a1c9515-bdee-4eb3-9198-39fcf3283ab3","Type":"ContainerStarted","Data":"eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a"}
Sep 09 10:47:03.616438 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:03.616041    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp" event={"ID":"0a1c9515-bdee-4eb3-9198-39fcf3283ab3","Type":"ContainerStarted","Data":"7b74c999b91b1bfdb5a17b85915da3a43c2766347bf1cbd1e0979766eba590d4"}
Sep 09 10:47:03.638759 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:03.638710    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp" podStartSLOduration=1.638694425 podStartE2EDuration="1.638694425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:47:03.636175602 +0000 UTC m=+1265.691734671" watchObservedRunningTime="2026-09-09 10:47:03.638694425 +0000 UTC m=+1265.694253474"
Sep 09 10:47:10.027953 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:10.027915    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:47:12.922566 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:12.922519    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:12.923025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:12.922586    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:12.935166 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:12.935142    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:13.695993 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:13.695966    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:47:18.489570 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:18.489536    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"]
Sep 09 10:47:18.490060 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:18.489747    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy" containerID="cri-o://6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d" gracePeriod=30
Sep 09 10:47:23.745356 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.745332    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:47:23.752609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.752549    2586 generic.go:356] "Generic (PLEG): container finished" podID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerID="6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d" exitCode=0
Sep 09 10:47:23.752609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.752602    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"
Sep 09 10:47:23.752756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.752609    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" event={"ID":"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f","Type":"ContainerDied","Data":"6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d"}
Sep 09 10:47:23.752756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.752680    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2" event={"ID":"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f","Type":"ContainerDied","Data":"5f78717eda24a731962e0c5dcae8dcb94194bb1ed592d6d48bc85af2ea110fa6"}
Sep 09 10:47:23.752756 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.752706    2586 scope.go:117] "RemoveContainer" containerID="6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d"
Sep 09 10:47:23.763771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.763750    2586 scope.go:117] "RemoveContainer" containerID="fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884"
Sep 09 10:47:23.772488 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.772473    2586 scope.go:117] "RemoveContainer" containerID="6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d"
Sep 09 10:47:23.772756 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:47:23.772738    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d\": container with ID starting with 6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d not found: ID does not exist" containerID="6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d"
Sep 09 10:47:23.772803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.772766    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d"} err="failed to get container status \"6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d\": rpc error: code = NotFound desc = could not find container \"6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d\": container with ID starting with 6882a0e355d7b90ff6f95afac02105ffabed6570bff224b0a83bc4b68d2bfc6d not found: ID does not exist"
Sep 09 10:47:23.772803 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.772783    2586 scope.go:117] "RemoveContainer" containerID="fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884"
Sep 09 10:47:23.773079 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:47:23.773061    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884\": container with ID starting with fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884 not found: ID does not exist" containerID="fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884"
Sep 09 10:47:23.773134 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.773084    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884"} err="failed to get container status \"fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884\": rpc error: code = NotFound desc = could not find container \"fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884\": container with ID starting with fe96250f6f48e73a118d6e3c37ce44c642b9b7871d0ac75c5ab83dd9d8f25884 not found: ID does not exist"
Sep 09 10:47:23.864195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864162    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sp9f\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-kube-api-access-5sp9f\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864197    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-envoy\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864457 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864220    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-data\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864457 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864282    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-certs\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864457 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864344    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-socket\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864457 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864388    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-token\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864457 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864423    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istiod-ca-cert\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864722 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864462    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-credential-socket\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864722 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864486    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-podinfo\") pod \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\" (UID: \"31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f\") "
Sep 09 10:47:23.864722 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864581    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:23.864722 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864604    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-data" (OuterVolumeSpecName: "istio-data") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:23.864722 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864636    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:23.865025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864776    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:23.865025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.864989    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:23.865025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.865012    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:23.865160 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.865027    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:23.865160 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.865039    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:23.865160 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.865077    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:47:23.866626 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.866602    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-token" (OuterVolumeSpecName: "istio-token") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:47:23.866725 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.866636    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-kube-api-access-5sp9f" (OuterVolumeSpecName: "kube-api-access-5sp9f") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "kube-api-access-5sp9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:47:23.866780 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.866758    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:47:23.866880 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.866857    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" (UID: "31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:47:23.966109 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.966075    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:23.966109 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.966103    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:23.966109 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.966112    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:23.966333 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.966120    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sp9f\" (UniqueName: \"kubernetes.io/projected/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-kube-api-access-5sp9f\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:23.966333 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:23.966132    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:47:24.079753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:24.079724    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"]
Sep 09 10:47:24.086220 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:24.086193    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4bbfe3-openshift-default-796bcfc4-j2xv2"]
Sep 09 10:47:24.476665 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:24.476630    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" path="/var/lib/kubelet/pods/31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f/volumes"
Sep 09 10:47:24.813205 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:24.813122    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:47:35.906414 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:35.906376    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:47:49.813232 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:47:49.813195    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:48:01.319346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.319313    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"]
Sep 09 10:48:01.319773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.319759    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy"
Sep 09 10:48:01.319773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.319774    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy"
Sep 09 10:48:01.319848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.319783    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy"
Sep 09 10:48:01.319848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.319789    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy"
Sep 09 10:48:01.319935 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.319860    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy"
Sep 09 10:48:01.319935 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.319867    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ff2e9a-29e4-4cdc-b6b4-f2e782e5e79f" containerName="istio-proxy"
Sep 09 10:48:01.324161 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.324142    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.327836 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.327816    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-928d2a\"/\"openshift-service-ca.crt\""
Sep 09 10:48:01.328105 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.328087    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-928d2a\"/\"explicit-model-928d2a-kserve-self-signed-certs\""
Sep 09 10:48:01.328374 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.328349    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-928d2a\"/\"kube-root-ca.crt\""
Sep 09 10:48:01.328683 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.328666    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-928d2a\"/\"default-dockercfg-brncz\""
Sep 09 10:48:01.335610 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.335584    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"]
Sep 09 10:48:01.430318 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.430281    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-dshm\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.430489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.430338    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.430489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.430357    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-model-cache\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.430489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.430419    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xctc9\" (UniqueName: \"kubernetes.io/projected/b02f1489-a4a9-4e25-b481-6c9955c706cb-kube-api-access-xctc9\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.430672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.430555    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-tmp-dir\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.430672 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.430662    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-home\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.531589 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.531554    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-dshm\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.531762 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.531694    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.531762 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.531713    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-model-cache\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.531762 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.531742    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xctc9\" (UniqueName: \"kubernetes.io/projected/b02f1489-a4a9-4e25-b481-6c9955c706cb-kube-api-access-xctc9\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.531927 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.531791    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-tmp-dir\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.531927 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.531874    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-home\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.532245 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.532218    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-model-cache\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.532369 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.532293    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-home\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.532369 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.532319    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-tmp-dir\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.534165 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.534140    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-dshm\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.534393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.534377    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.541620 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.541596    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xctc9\" (UniqueName: \"kubernetes.io/projected/b02f1489-a4a9-4e25-b481-6c9955c706cb-kube-api-access-xctc9\") pod \"explicit-model-928d2a-kserve-859b7c94d5-z2tw8\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") " pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.634542 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.634511    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:01.909409 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.909376    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"]
Sep 09 10:48:01.910613 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:48:01.910585    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb02f1489_a4a9_4e25_b481_6c9955c706cb.slice/crio-199743d78d1fbb9c469269e92553084711382e3b83f9430ad810d1a9f462b6c0 WatchSource:0}: Error finding container 199743d78d1fbb9c469269e92553084711382e3b83f9430ad810d1a9f462b6c0: Status 404 returned error can't find the container with id 199743d78d1fbb9c469269e92553084711382e3b83f9430ad810d1a9f462b6c0
Sep 09 10:48:01.988732 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:01.988700    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8" event={"ID":"b02f1489-a4a9-4e25-b481-6c9955c706cb","Type":"ContainerStarted","Data":"199743d78d1fbb9c469269e92553084711382e3b83f9430ad810d1a9f462b6c0"}
Sep 09 10:48:02.997063 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:02.997023    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8" event={"ID":"b02f1489-a4a9-4e25-b481-6c9955c706cb","Type":"ContainerStarted","Data":"ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b"}
Sep 09 10:48:03.020919 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:03.020839    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8" podStartSLOduration=2.020819242 podStartE2EDuration="2.020819242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:03.019352849 +0000 UTC m=+1325.074911898" watchObservedRunningTime="2026-09-09 10:48:03.020819242 +0000 UTC m=+1325.076378293"
Sep 09 10:48:03.655274 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:03.655239    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"]
Sep 09 10:48:04.156753 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:04.156719    2586 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-self-signed-certs: secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:04.157146 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:04.156789    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs podName:b02f1489-a4a9-4e25-b481-6c9955c706cb nodeName:}" failed. No retries permitted until 2026-09-09 10:48:04.6567741 +0000 UTC m=+1326.712333128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs") pod "explicit-model-928d2a-kserve-859b7c94d5-z2tw8" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb") : secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:04.662500 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:04.662464    2586 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-self-signed-certs: secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:04.662684 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:04.662543    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs podName:b02f1489-a4a9-4e25-b481-6c9955c706cb nodeName:}" failed. No retries permitted until 2026-09-09 10:48:05.662524784 +0000 UTC m=+1327.718083813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs") pod "explicit-model-928d2a-kserve-859b7c94d5-z2tw8" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb") : secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:05.131411 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:05.131373    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:48:05.673708 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:05.673676    2586 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-self-signed-certs: secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:05.674141 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:05.673745    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs podName:b02f1489-a4a9-4e25-b481-6c9955c706cb nodeName:}" failed. No retries permitted until 2026-09-09 10:48:07.673731322 +0000 UTC m=+1329.729290350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs") pod "explicit-model-928d2a-kserve-859b7c94d5-z2tw8" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb") : secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:07.698282 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:07.698245    2586 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-self-signed-certs: secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:07.698672 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:07.698331    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs podName:b02f1489-a4a9-4e25-b481-6c9955c706cb nodeName:}" failed. No retries permitted until 2026-09-09 10:48:11.698315653 +0000 UTC m=+1333.753874681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs") pod "explicit-model-928d2a-kserve-859b7c94d5-z2tw8" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb") : secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:11.330346 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:11.330310    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"]
Sep 09 10:48:11.635163 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:11.635117    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:11.740461 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:11.740428    2586 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-self-signed-certs: secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:11.740633 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:11.740503    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs podName:b02f1489-a4a9-4e25-b481-6c9955c706cb nodeName:}" failed. No retries permitted until 2026-09-09 10:48:19.740488949 +0000 UTC m=+1341.796047978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs") pod "explicit-model-928d2a-kserve-859b7c94d5-z2tw8" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb") : secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:19.820291 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:19.820256    2586 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-self-signed-certs: secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:19.820697 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:19.820338    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs podName:b02f1489-a4a9-4e25-b481-6c9955c706cb nodeName:}" failed. No retries permitted until 2026-09-09 10:48:35.820322439 +0000 UTC m=+1357.875881468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs") pod "explicit-model-928d2a-kserve-859b7c94d5-z2tw8" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb") : secret "explicit-model-928d2a-kserve-self-signed-certs" not found
Sep 09 10:48:20.028869 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.028810    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8" podUID="b02f1489-a4a9-4e25-b481-6c9955c706cb" containerName="main" containerID="cri-o://ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b" gracePeriod=45
Sep 09 10:48:20.284274 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.284250    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:20.324089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324058    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-home\") pod \"b02f1489-a4a9-4e25-b481-6c9955c706cb\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") "
Sep 09 10:48:20.324237 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324109    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs\") pod \"b02f1489-a4a9-4e25-b481-6c9955c706cb\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") "
Sep 09 10:48:20.324237 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324138    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-model-cache\") pod \"b02f1489-a4a9-4e25-b481-6c9955c706cb\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") "
Sep 09 10:48:20.324237 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324207    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-dshm\") pod \"b02f1489-a4a9-4e25-b481-6c9955c706cb\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") "
Sep 09 10:48:20.324416 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324264    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-tmp-dir\") pod \"b02f1489-a4a9-4e25-b481-6c9955c706cb\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") "
Sep 09 10:48:20.324416 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324330    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-home" (OuterVolumeSpecName: "home") pod "b02f1489-a4a9-4e25-b481-6c9955c706cb" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:20.324416 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324351    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xctc9\" (UniqueName: \"kubernetes.io/projected/b02f1489-a4a9-4e25-b481-6c9955c706cb-kube-api-access-xctc9\") pod \"b02f1489-a4a9-4e25-b481-6c9955c706cb\" (UID: \"b02f1489-a4a9-4e25-b481-6c9955c706cb\") "
Sep 09 10:48:20.324416 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324399    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "b02f1489-a4a9-4e25-b481-6c9955c706cb" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:20.324604 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324537    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b02f1489-a4a9-4e25-b481-6c9955c706cb" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:20.324687 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324671    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:20.324741 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324692    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:20.324741 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.324704    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:20.326384 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.326358    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-dshm" (OuterVolumeSpecName: "dshm") pod "b02f1489-a4a9-4e25-b481-6c9955c706cb" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:20.326521 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.326435    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b02f1489-a4a9-4e25-b481-6c9955c706cb" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:20.326521 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.326459    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02f1489-a4a9-4e25-b481-6c9955c706cb-kube-api-access-xctc9" (OuterVolumeSpecName: "kube-api-access-xctc9") pod "b02f1489-a4a9-4e25-b481-6c9955c706cb" (UID: "b02f1489-a4a9-4e25-b481-6c9955c706cb"). InnerVolumeSpecName "kube-api-access-xctc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:20.425992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.425920    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02f1489-a4a9-4e25-b481-6c9955c706cb-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:20.425992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.425945    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xctc9\" (UniqueName: \"kubernetes.io/projected/b02f1489-a4a9-4e25-b481-6c9955c706cb-kube-api-access-xctc9\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:20.425992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:20.425958    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02f1489-a4a9-4e25-b481-6c9955c706cb-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:21.110842 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.110811    2586 generic.go:356] "Generic (PLEG): container finished" podID="b02f1489-a4a9-4e25-b481-6c9955c706cb" containerID="ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b" exitCode=0
Sep 09 10:48:21.111362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.110857    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"
Sep 09 10:48:21.111362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.110913    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8" event={"ID":"b02f1489-a4a9-4e25-b481-6c9955c706cb","Type":"ContainerDied","Data":"ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b"}
Sep 09 10:48:21.111362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.110955    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8" event={"ID":"b02f1489-a4a9-4e25-b481-6c9955c706cb","Type":"ContainerDied","Data":"199743d78d1fbb9c469269e92553084711382e3b83f9430ad810d1a9f462b6c0"}
Sep 09 10:48:21.111362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.110974    2586 scope.go:117] "RemoveContainer" containerID="ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b"
Sep 09 10:48:21.120104 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.120086    2586 scope.go:117] "RemoveContainer" containerID="ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b"
Sep 09 10:48:21.120385 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:21.120365    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b\": container with ID starting with ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b not found: ID does not exist" containerID="ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b"
Sep 09 10:48:21.120426 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.120394    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b"} err="failed to get container status \"ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b\": rpc error: code = NotFound desc = could not find container \"ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b\": container with ID starting with ea34846abda93ee25278615f7f8a53d95126195afe331c4494cb54a90259096b not found: ID does not exist"
Sep 09 10:48:21.127221 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.127201    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"]
Sep 09 10:48:21.131135 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:21.131111    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-928d2a/explicit-model-928d2a-kserve-859b7c94d5-z2tw8"]
Sep 09 10:48:22.307280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:22.307244    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:48:22.368527 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:22.368497    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"]
Sep 09 10:48:22.371924 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:22.371883    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40f1e9-29815845-4fhkt"]
Sep 09 10:48:22.474868 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:22.474833    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14290167-912d-43f5-84cf-fc677d2fd768" path="/var/lib/kubelet/pods/14290167-912d-43f5-84cf-fc677d2fd768/volumes"
Sep 09 10:48:22.475243 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:22.475228    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02f1489-a4a9-4e25-b481-6c9955c706cb" path="/var/lib/kubelet/pods/b02f1489-a4a9-4e25-b481-6c9955c706cb/volumes"
Sep 09 10:48:26.351010 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.350966    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp" podUID="0a1c9515-bdee-4eb3-9198-39fcf3283ab3" containerName="main" containerID="cri-o://eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a" gracePeriod=45
Sep 09 10:48:26.592351 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.592318    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:48:26.690852 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.690820    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-dshm\") pod \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") "
Sep 09 10:48:26.691028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.690866    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-model-cache\") pod \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") "
Sep 09 10:48:26.691028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691018    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tls-certs\") pod \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") "
Sep 09 10:48:26.691158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691051    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-home\") pod \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") "
Sep 09 10:48:26.691158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691092    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkzhj\" (UniqueName: \"kubernetes.io/projected/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-kube-api-access-hkzhj\") pod \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") "
Sep 09 10:48:26.691158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691119    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tmp-dir\") pod \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\" (UID: \"0a1c9515-bdee-4eb3-9198-39fcf3283ab3\") "
Sep 09 10:48:26.691310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691215    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a1c9515-bdee-4eb3-9198-39fcf3283ab3" (UID: "0a1c9515-bdee-4eb3-9198-39fcf3283ab3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:26.691365 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691325    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-home" (OuterVolumeSpecName: "home") pod "0a1c9515-bdee-4eb3-9198-39fcf3283ab3" (UID: "0a1c9515-bdee-4eb3-9198-39fcf3283ab3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:26.691505 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691474    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a1c9515-bdee-4eb3-9198-39fcf3283ab3" (UID: "0a1c9515-bdee-4eb3-9198-39fcf3283ab3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:26.691587 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691565    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:26.691645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.691586    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:26.693108 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.693084    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-dshm" (OuterVolumeSpecName: "dshm") pod "0a1c9515-bdee-4eb3-9198-39fcf3283ab3" (UID: "0a1c9515-bdee-4eb3-9198-39fcf3283ab3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:26.693217 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.693195    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-kube-api-access-hkzhj" (OuterVolumeSpecName: "kube-api-access-hkzhj") pod "0a1c9515-bdee-4eb3-9198-39fcf3283ab3" (UID: "0a1c9515-bdee-4eb3-9198-39fcf3283ab3"). InnerVolumeSpecName "kube-api-access-hkzhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:26.693267 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.693221    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a1c9515-bdee-4eb3-9198-39fcf3283ab3" (UID: "0a1c9515-bdee-4eb3-9198-39fcf3283ab3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:26.793128 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.793094    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:26.793128 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.793124    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkzhj\" (UniqueName: \"kubernetes.io/projected/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-kube-api-access-hkzhj\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:26.793282 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.793135    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:26.793282 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:26.793146    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a1c9515-bdee-4eb3-9198-39fcf3283ab3-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:27.151002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.150962    2586 generic.go:356] "Generic (PLEG): container finished" podID="0a1c9515-bdee-4eb3-9198-39fcf3283ab3" containerID="eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a" exitCode=0
Sep 09 10:48:27.151002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.151004    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"
Sep 09 10:48:27.151248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.151049    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp" event={"ID":"0a1c9515-bdee-4eb3-9198-39fcf3283ab3","Type":"ContainerDied","Data":"eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a"}
Sep 09 10:48:27.151248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.151096    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp" event={"ID":"0a1c9515-bdee-4eb3-9198-39fcf3283ab3","Type":"ContainerDied","Data":"7b74c999b91b1bfdb5a17b85915da3a43c2766347bf1cbd1e0979766eba590d4"}
Sep 09 10:48:27.151248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.151115    2586 scope.go:117] "RemoveContainer" containerID="eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a"
Sep 09 10:48:27.160952 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.160933    2586 scope.go:117] "RemoveContainer" containerID="eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a"
Sep 09 10:48:27.161244 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:27.161200    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a\": container with ID starting with eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a not found: ID does not exist" containerID="eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a"
Sep 09 10:48:27.161244 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.161229    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a"} err="failed to get container status \"eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a\": rpc error: code = NotFound desc = could not find container \"eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a\": container with ID starting with eb21933f86cb7845418fdf8458bd4cf4cb3866771f1ff8431f09533149cae97a not found: ID does not exist"
Sep 09 10:48:27.172194 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.172174    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"]
Sep 09 10:48:27.176232 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:27.176205    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-da3c64/test-model-da3c64-kserve-7f878b7779-4jhbp"]
Sep 09 10:48:28.476996 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:28.476965    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a1c9515-bdee-4eb3-9198-39fcf3283ab3" path="/var/lib/kubelet/pods/0a1c9515-bdee-4eb3-9198-39fcf3283ab3/volumes"
Sep 09 10:48:30.805076 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:30.805036    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:48:39.407201 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:39.407165    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:48:40.552424 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.552385    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"]
Sep 09 10:48:40.552853 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.552617    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb" podUID="7a758eed-a041-49b2-84f4-54b2803f495f" containerName="maas-api" containerID="cri-o://38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b" gracePeriod=30
Sep 09 10:48:40.804745 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.804679    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:48:40.937678 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.937649    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls\") pod \"7a758eed-a041-49b2-84f4-54b2803f495f\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") "
Sep 09 10:48:40.937882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.937717    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlgl8\" (UniqueName: \"kubernetes.io/projected/7a758eed-a041-49b2-84f4-54b2803f495f-kube-api-access-tlgl8\") pod \"7a758eed-a041-49b2-84f4-54b2803f495f\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") "
Sep 09 10:48:40.937882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.937746    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a758eed-a041-49b2-84f4-54b2803f495f-openshift-service-ca\") pod \"7a758eed-a041-49b2-84f4-54b2803f495f\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") "
Sep 09 10:48:40.937882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.937778    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-metrics-certs\") pod \"7a758eed-a041-49b2-84f4-54b2803f495f\" (UID: \"7a758eed-a041-49b2-84f4-54b2803f495f\") "
Sep 09 10:48:40.938415 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.938383    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a758eed-a041-49b2-84f4-54b2803f495f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a758eed-a041-49b2-84f4-54b2803f495f" (UID: "7a758eed-a041-49b2-84f4-54b2803f495f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:48:40.940093 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.940062    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a758eed-a041-49b2-84f4-54b2803f495f" (UID: "7a758eed-a041-49b2-84f4-54b2803f495f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:40.940256 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.940236    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a758eed-a041-49b2-84f4-54b2803f495f-kube-api-access-tlgl8" (OuterVolumeSpecName: "kube-api-access-tlgl8") pod "7a758eed-a041-49b2-84f4-54b2803f495f" (UID: "7a758eed-a041-49b2-84f4-54b2803f495f"). InnerVolumeSpecName "kube-api-access-tlgl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:40.940616 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:40.940579    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a758eed-a041-49b2-84f4-54b2803f495f" (UID: "7a758eed-a041-49b2-84f4-54b2803f495f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:48:41.039352 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.039307    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:41.039352 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.039348    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlgl8\" (UniqueName: \"kubernetes.io/projected/7a758eed-a041-49b2-84f4-54b2803f495f-kube-api-access-tlgl8\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:41.039352 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.039358    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a758eed-a041-49b2-84f4-54b2803f495f-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:41.039600 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.039367    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a758eed-a041-49b2-84f4-54b2803f495f-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:41.238493 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.238457    2586 generic.go:356] "Generic (PLEG): container finished" podID="7a758eed-a041-49b2-84f4-54b2803f495f" containerID="38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b" exitCode=0
Sep 09 10:48:41.238675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.238504    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"
Sep 09 10:48:41.238675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.238550    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb" event={"ID":"7a758eed-a041-49b2-84f4-54b2803f495f","Type":"ContainerDied","Data":"38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b"}
Sep 09 10:48:41.238675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.238610    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb" event={"ID":"7a758eed-a041-49b2-84f4-54b2803f495f","Type":"ContainerDied","Data":"b5404891de4199a2b4e6c2a3db447b5e3b4a3c1af77aed44bc5cb046466c82fa"}
Sep 09 10:48:41.238675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.238637    2586 scope.go:117] "RemoveContainer" containerID="38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b"
Sep 09 10:48:41.251453 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.251428    2586 scope.go:117] "RemoveContainer" containerID="38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b"
Sep 09 10:48:41.251926 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:41.251880    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b\": container with ID starting with 38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b not found: ID does not exist" containerID="38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b"
Sep 09 10:48:41.251997 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.251939    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b"} err="failed to get container status \"38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b\": rpc error: code = NotFound desc = could not find container \"38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b\": container with ID starting with 38d48b9a3c01f6873eef7207b3d696aaa715477b0e45bb25f06dbbdb3389d02b not found: ID does not exist"
Sep 09 10:48:41.263741 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.263704    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"]
Sep 09 10:48:41.266064 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:41.266041    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da3c64-5d7997bcb-78jdb"]
Sep 09 10:48:42.474966 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:42.474933    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a758eed-a041-49b2-84f4-54b2803f495f" path="/var/lib/kubelet/pods/7a758eed-a041-49b2-84f4-54b2803f495f/volumes"
Sep 09 10:48:46.906404 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:46.906369    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:48:47.695091 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:47.695056    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"]
Sep 09 10:48:47.695377 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:47.695350    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" containerID="cri-o://70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c" gracePeriod=30
Sep 09 10:48:50.111814 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:50.111777    2586 patch_prober.go:28] interesting pod/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 10:48:50.112266 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:50.111864    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:48:51.130540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.130506    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"]
Sep 09 10:48:51.131126 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131105    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a1c9515-bdee-4eb3-9198-39fcf3283ab3" containerName="main"
Sep 09 10:48:51.131224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131128    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a1c9515-bdee-4eb3-9198-39fcf3283ab3" containerName="main"
Sep 09 10:48:51.131224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131164    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a758eed-a041-49b2-84f4-54b2803f495f" containerName="maas-api"
Sep 09 10:48:51.131224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131174    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a758eed-a041-49b2-84f4-54b2803f495f" containerName="maas-api"
Sep 09 10:48:51.131224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131189    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02f1489-a4a9-4e25-b481-6c9955c706cb" containerName="main"
Sep 09 10:48:51.131224 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131197    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02f1489-a4a9-4e25-b481-6c9955c706cb" containerName="main"
Sep 09 10:48:51.131461 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131303    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02f1489-a4a9-4e25-b481-6c9955c706cb" containerName="main"
Sep 09 10:48:51.131461 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131320    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a758eed-a041-49b2-84f4-54b2803f495f" containerName="maas-api"
Sep 09 10:48:51.131461 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.131330    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a1c9515-bdee-4eb3-9198-39fcf3283ab3" containerName="main"
Sep 09 10:48:51.135863 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.135841    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.137975 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.137950    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2a090f42-openshift-default-dockercfg-cbkdk\""
Sep 09 10:48:51.150509 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.150480    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"]
Sep 09 10:48:51.231603 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231563    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-credential-socket\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.231603 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231615    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-socket\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.231838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231681    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020c1159-38bc-4914-b204-c3d7f1f2ca49-istiod-ca-cert\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.231838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231716    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-envoy\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.231838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231738    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-certs\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.231838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231777    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-data\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.231838 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231829    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6vp\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-kube-api-access-sk6vp\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.232073 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231847    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-podinfo\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.232073 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.231923    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-token\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.332878 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.332834    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-envoy\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.332878 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.332879    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-certs\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.332943    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-data\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.332970    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6vp\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-kube-api-access-sk6vp\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.332986    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-podinfo\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-token\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-credential-socket\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333124    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-socket\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333170    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020c1159-38bc-4914-b204-c3d7f1f2ca49-istiod-ca-cert\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333614 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-data\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333614 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333429    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-certs\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333738 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333668    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-credential-socket\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333852 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333825    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020c1159-38bc-4914-b204-c3d7f1f2ca49-istiod-ca-cert\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.333996 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.333912    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-socket\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.335285 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.335263    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-envoy\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.335505 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.335485    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-podinfo\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.340802 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.340772    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-token\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.341053 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.341033    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6vp\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-kube-api-access-sk6vp\") pod \"e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") " pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.447017 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.446923    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:51.738345 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.738313    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"]
Sep 09 10:48:51.738474 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:48:51.738398    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020c1159_38bc_4914_b204_c3d7f1f2ca49.slice/crio-cbe5e03b5d0a971260f7d824ade3a3beb68762e8425d76e0dfaf5b4f88c94923 WatchSource:0}: Error finding container cbe5e03b5d0a971260f7d824ade3a3beb68762e8425d76e0dfaf5b4f88c94923: Status 404 returned error can't find the container with id cbe5e03b5d0a971260f7d824ade3a3beb68762e8425d76e0dfaf5b4f88c94923
Sep 09 10:48:51.740693 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.740659    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:51.740787 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.740735    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:51.740787 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:51.740777    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:48:52.305412 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:52.305377    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" event={"ID":"020c1159-38bc-4914-b204-c3d7f1f2ca49","Type":"ContainerStarted","Data":"4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd"}
Sep 09 10:48:52.305879 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:52.305430    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" event={"ID":"020c1159-38bc-4914-b204-c3d7f1f2ca49","Type":"ContainerStarted","Data":"cbe5e03b5d0a971260f7d824ade3a3beb68762e8425d76e0dfaf5b4f88c94923"}
Sep 09 10:48:52.331017 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:52.330963    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" podStartSLOduration=1.330947547 podStartE2EDuration="1.330947547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:52.327835455 +0000 UTC m=+1374.383394518" watchObservedRunningTime="2026-09-09 10:48:52.330947547 +0000 UTC m=+1374.386506597"
Sep 09 10:48:52.447709 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:52.447647    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:52.452796 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:52.452766    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:52.955707 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:52.955643    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:48:53.016589 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.016545    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"]
Sep 09 10:48:53.017012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.016996    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy"
Sep 09 10:48:53.017012 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.017011    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy"
Sep 09 10:48:53.017159 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.017085    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy"
Sep 09 10:48:53.017159 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.017092    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy"
Sep 09 10:48:53.020361 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.020333    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.022429 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.022399    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1eb21d-serving-cert\""
Sep 09 10:48:53.022616 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.022451    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k8bm4\""
Sep 09 10:48:53.022916 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.022744    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 10:48:53.022916 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.022839    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 10:48:53.022916 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.022856    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 10:48:53.035115 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.035088    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"]
Sep 09 10:48:53.048098 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048058    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-socket\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048240 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048137    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-envoy\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048240 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048164    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-certs\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048240 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048193    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-token\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048240 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048217    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be21e8f-f131-42bd-934d-b3be0cf1c879-istiod-ca-cert\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048474 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048273    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkx9g\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-kube-api-access-tkx9g\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048474 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048324    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-podinfo\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048474 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048365    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-data\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048474 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048368    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:53.048474 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048398    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-credential-socket\") pod \"0be21e8f-f131-42bd-934d-b3be0cf1c879\" (UID: \"0be21e8f-f131-42bd-934d-b3be0cf1c879\") "
Sep 09 10:48:53.048725 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048492    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:53.048725 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048584    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.048725 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048674    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b777b12-6493-4377-9ef3-7617c69381a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.048725 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048694    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:53.048725 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048704    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j722l\" (UniqueName: \"kubernetes.io/projected/2b777b12-6493-4377-9ef3-7617c69381a1-kube-api-access-j722l\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.049003 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048884    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.049003 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048978    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.049003 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.048994    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.049171 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.049008    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.049171 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.049117    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-data" (OuterVolumeSpecName: "istio-data") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:53.049278 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.049233    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be21e8f-f131-42bd-934d-b3be0cf1c879-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:48:53.051085 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.051057    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:48:53.051267 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.051245    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-token" (OuterVolumeSpecName: "istio-token") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:53.051633 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.051609    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:48:53.051733 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.051616    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-kube-api-access-tkx9g" (OuterVolumeSpecName: "kube-api-access-tkx9g") pod "0be21e8f-f131-42bd-934d-b3be0cf1c879" (UID: "0be21e8f-f131-42bd-934d-b3be0cf1c879"). InnerVolumeSpecName "kube-api-access-tkx9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:48:53.150496 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150453    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.150687 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150512    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.150687 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150547    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b777b12-6493-4377-9ef3-7617c69381a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.150687 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150566    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j722l\" (UniqueName: \"kubernetes.io/projected/2b777b12-6493-4377-9ef3-7617c69381a1-kube-api-access-j722l\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.150687 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150648    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkx9g\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-kube-api-access-tkx9g\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.150687 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150666    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.150687 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150680    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.151042 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:53.150688    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-serving-cert: secret "maas-api-e2e-shared-a-w3-1eb21d-serving-cert" not found
Sep 09 10:48:53.151042 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:53.150760    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls podName:2b777b12-6493-4377-9ef3-7617c69381a1 nodeName:}" failed. No retries permitted until 2026-09-09 10:48:53.650743822 +0000 UTC m=+1375.706302851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls") pod "maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9" (UID: "2b777b12-6493-4377-9ef3-7617c69381a1") : secret "maas-api-e2e-shared-a-w3-1eb21d-serving-cert" not found
Sep 09 10:48:53.151042 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150694    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.151042 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150799    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be21e8f-f131-42bd-934d-b3be0cf1c879-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.151042 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.150813    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be21e8f-f131-42bd-934d-b3be0cf1c879-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:48:53.151433 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.151408    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b777b12-6493-4377-9ef3-7617c69381a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.153118 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.153094    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.158644 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.158619    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j722l\" (UniqueName: \"kubernetes.io/projected/2b777b12-6493-4377-9ef3-7617c69381a1-kube-api-access-j722l\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.313271 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.313180    2586 generic.go:356] "Generic (PLEG): container finished" podID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerID="70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c" exitCode=0
Sep 09 10:48:53.313271 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.313234    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"
Sep 09 10:48:53.313781 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.313268    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" event={"ID":"0be21e8f-f131-42bd-934d-b3be0cf1c879","Type":"ContainerDied","Data":"70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c"}
Sep 09 10:48:53.313781 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.313330    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z" event={"ID":"0be21e8f-f131-42bd-934d-b3be0cf1c879","Type":"ContainerDied","Data":"aa83f052b32270a161aa2cfe5f32f05840854ed9f6695a7daa84bb51b07ca4d3"}
Sep 09 10:48:53.313781 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.313361    2586 scope.go:117] "RemoveContainer" containerID="70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c"
Sep 09 10:48:53.313781 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.313696    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:53.314761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.314750    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:48:53.323683 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.323667    2586 scope.go:117] "RemoveContainer" containerID="11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff"
Sep 09 10:48:53.332589 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.332570    2586 scope.go:117] "RemoveContainer" containerID="70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c"
Sep 09 10:48:53.332937 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:53.332886    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c\": container with ID starting with 70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c not found: ID does not exist" containerID="70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c"
Sep 09 10:48:53.332993 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.332951    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c"} err="failed to get container status \"70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c\": rpc error: code = NotFound desc = could not find container \"70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c\": container with ID starting with 70d826ba604cd91a1892070e2141f3598573e8cf3a2505c8fadae09cc7a1613c not found: ID does not exist"
Sep 09 10:48:53.332993 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.332980    2586 scope.go:117] "RemoveContainer" containerID="11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff"
Sep 09 10:48:53.333240 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:48:53.333222    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff\": container with ID starting with 11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff not found: ID does not exist" containerID="11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff"
Sep 09 10:48:53.333296 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.333248    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff"} err="failed to get container status \"11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff\": rpc error: code = NotFound desc = could not find container \"11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff\": container with ID starting with 11d85ce8eb6dc37b9ccb03d267d0304fbb9bca5f0fab0c3ba3e8be1c1f4f46ff not found: ID does not exist"
Sep 09 10:48:53.352785 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.352751    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"]
Sep 09 10:48:53.356557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.356531    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-da3c64-openshift-default-644cc9d87d-r5m6z"]
Sep 09 10:48:53.655974 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.655938    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.658493 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.658464    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:53.933615 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:53.933521    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:54.210620 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:54.207526    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"]
Sep 09 10:48:54.212692 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:48:54.212657    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b777b12_6493_4377_9ef3_7617c69381a1.slice/crio-61a07c6d06138f6d7ca868ce714fe8936fc4bef8fc6187699c27959c7837802e WatchSource:0}: Error finding container 61a07c6d06138f6d7ca868ce714fe8936fc4bef8fc6187699c27959c7837802e: Status 404 returned error can't find the container with id 61a07c6d06138f6d7ca868ce714fe8936fc4bef8fc6187699c27959c7837802e
Sep 09 10:48:54.214487 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:54.214469    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:48:54.320549 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:54.320493    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9" event={"ID":"2b777b12-6493-4377-9ef3-7617c69381a1","Type":"ContainerStarted","Data":"61a07c6d06138f6d7ca868ce714fe8936fc4bef8fc6187699c27959c7837802e"}
Sep 09 10:48:54.475390 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:54.475310    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" path="/var/lib/kubelet/pods/0be21e8f-f131-42bd-934d-b3be0cf1c879/volumes"
Sep 09 10:48:55.330447 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:55.330411    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9" event={"ID":"2b777b12-6493-4377-9ef3-7617c69381a1","Type":"ContainerStarted","Data":"073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3"}
Sep 09 10:48:55.330951 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:55.330545    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:48:55.349180 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:55.349126    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9" podStartSLOduration=2.883569683 podStartE2EDuration="3.349109539s" totalImagesPullingTime="465.539856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:55.347269231 +0000 UTC m=+1377.402828300" watchObservedRunningTime="2026-09-09 10:48:55.349109539 +0000 UTC m=+1377.404668589"
Sep 09 10:48:56.404361 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:48:56.404327    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:49:01.341634 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:01.341601    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 10:49:06.898262 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:06.898222    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:49:12.436958 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:12.436920    2586 generic.go:356] "Generic (PLEG): container finished" podID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerID="4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd" exitCode=0
Sep 09 10:49:12.437396 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:12.436979    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" event={"ID":"020c1159-38bc-4914-b204-c3d7f1f2ca49","Type":"ContainerDied","Data":"4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd"}
Sep 09 10:49:12.437441 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:12.437418    2586 scope.go:117] "RemoveContainer" containerID="4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd"
Sep 09 10:49:12.438529 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:12.438499    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:12.438633 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:12.438585    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:12.438697 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:12.438634    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:49:13.449467 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:13.449426    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" event={"ID":"020c1159-38bc-4914-b204-c3d7f1f2ca49","Type":"ContainerStarted","Data":"3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c"}
Sep 09 10:49:14.447389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:14.447339    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:49:14.447795 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:14.447764    2586 patch_prober.go:28] interesting pod/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 09 10:49:14.447927 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:14.447828    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 09 10:49:15.448150 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:15.448101    2586 patch_prober.go:28] interesting pod/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 09 10:49:15.448680 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:15.448194    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 09 10:49:16.448143 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:16.447669    2586 patch_prober.go:28] interesting pod/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 09 10:49:16.448143 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:16.447735    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 09 10:49:17.448386 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:17.448347    2586 patch_prober.go:28] interesting pod/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 09 10:49:17.448980 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:17.448948    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 09 10:49:18.592254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:18.592220    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:49:18.592674 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:18.592642    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:49:18.594843 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:18.594817    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:49:27.808927 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:27.808873    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:49:40.817808 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:40.817773    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:49:44.685628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:44.685593    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"]
Sep 09 10:49:44.686086 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:44.685832    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy" containerID="cri-o://3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c" gracePeriod=30
Sep 09 10:49:48.595737 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:48.595699    2586 patch_prober.go:28] interesting pod/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 10:49:48.596122 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:48.595773    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:49:49.931674 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:49.931646    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:49:50.001231 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001153    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-envoy\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001231 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001188    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-data\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001244    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-socket\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001261    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020c1159-38bc-4914-b204-c3d7f1f2ca49-istiod-ca-cert\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001287    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk6vp\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-kube-api-access-sk6vp\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001363    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-podinfo\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001389    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-credential-socket\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001423 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001415    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-certs\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001739 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001543    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-data" (OuterVolumeSpecName: "istio-data") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:50.001739 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001564    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:50.001739 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001628    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-token\") pod \"020c1159-38bc-4914-b204-c3d7f1f2ca49\" (UID: \"020c1159-38bc-4914-b204-c3d7f1f2ca49\") "
Sep 09 10:49:50.001739 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001681    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:50.002009 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001730    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:50.002009 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001753    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020c1159-38bc-4914-b204-c3d7f1f2ca49-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:49:50.002009 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.001997    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.002168 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.002015    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.002168 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.002024    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020c1159-38bc-4914-b204-c3d7f1f2ca49-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.002168 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.002032    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.002168 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.002041    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.003411 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.003383    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:49:50.003753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.003735    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:49:50.003881 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.003866    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-token" (OuterVolumeSpecName: "istio-token") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:49:50.003956 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.003902    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-kube-api-access-sk6vp" (OuterVolumeSpecName: "kube-api-access-sk6vp") pod "020c1159-38bc-4914-b204-c3d7f1f2ca49" (UID: "020c1159-38bc-4914-b204-c3d7f1f2ca49"). InnerVolumeSpecName "kube-api-access-sk6vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:49:50.102773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.102735    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.102773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.102768    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.102773 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.102777    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020c1159-38bc-4914-b204-c3d7f1f2ca49-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.103050 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.102786    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk6vp\" (UniqueName: \"kubernetes.io/projected/020c1159-38bc-4914-b204-c3d7f1f2ca49-kube-api-access-sk6vp\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:49:50.672904 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.672820    2586 generic.go:356] "Generic (PLEG): container finished" podID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerID="3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c" exitCode=0
Sep 09 10:49:50.672904 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.672869    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"
Sep 09 10:49:50.673082 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.672913    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" event={"ID":"020c1159-38bc-4914-b204-c3d7f1f2ca49","Type":"ContainerDied","Data":"3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c"}
Sep 09 10:49:50.673082 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.672970    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw" event={"ID":"020c1159-38bc-4914-b204-c3d7f1f2ca49","Type":"ContainerDied","Data":"cbe5e03b5d0a971260f7d824ade3a3beb68762e8425d76e0dfaf5b4f88c94923"}
Sep 09 10:49:50.673082 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.673001    2586 scope.go:117] "RemoveContainer" containerID="3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c"
Sep 09 10:49:50.682482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.682456    2586 scope.go:117] "RemoveContainer" containerID="4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd"
Sep 09 10:49:50.691433 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.691353    2586 scope.go:117] "RemoveContainer" containerID="3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c"
Sep 09 10:49:50.692064 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:49:50.691793    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c\": container with ID starting with 3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c not found: ID does not exist" containerID="3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c"
Sep 09 10:49:50.692064 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.691827    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c"} err="failed to get container status \"3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c\": rpc error: code = NotFound desc = could not find container \"3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c\": container with ID starting with 3c9487a203e26bc5008d3db2acca9454a00669436922801fa3a0b9315b9f732c not found: ID does not exist"
Sep 09 10:49:50.692064 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.691851    2586 scope.go:117] "RemoveContainer" containerID="4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd"
Sep 09 10:49:50.692402 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:49:50.692352    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd\": container with ID starting with 4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd not found: ID does not exist" containerID="4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd"
Sep 09 10:49:50.692402 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.692379    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd"} err="failed to get container status \"4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd\": rpc error: code = NotFound desc = could not find container \"4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd\": container with ID starting with 4c7b7813af850ea331190f7f5af041149d05fd4845f19463738516d4100524bd not found: ID does not exist"
Sep 09 10:49:50.693720 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.693699    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"]
Sep 09 10:49:50.698561 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:50.698541    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2a090f42-openshift-default-7844d76cdd-ssdsw"]
Sep 09 10:49:52.475377 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:52.475344    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" path="/var/lib/kubelet/pods/020c1159-38bc-4914-b204-c3d7f1f2ca49/volumes"
Sep 09 10:49:54.023172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:54.023128    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:49:59.889364 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.889332    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"]
Sep 09 10:49:59.889821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.889754    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy"
Sep 09 10:49:59.889821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.889768    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy"
Sep 09 10:49:59.889821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.889787    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy"
Sep 09 10:49:59.889821 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.889792    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be21e8f-f131-42bd-934d-b3be0cf1c879" containerName="istio-proxy"
Sep 09 10:49:59.890001 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.889864    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy"
Sep 09 10:49:59.890001 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.889875    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy"
Sep 09 10:49:59.893978 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.893958    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.896199 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.896176    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-62f893-openshift-default-dockercfg-x2trn\""
Sep 09 10:49:59.907495 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.907467    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"]
Sep 09 10:49:59.999093 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999056    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhtp\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-kube-api-access-jbhtp\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.999254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999123    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-socket\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.999254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999157    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-envoy\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.999254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999173    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-data\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.999254 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999194    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-credential-socket\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.999403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999285    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-certs\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.999403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999317    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-token\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.999403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999372    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-podinfo\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:49:59.999403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:49:59.999390    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6152dd-f966-4d23-9d91-42ab3743b64a-istiod-ca-cert\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.100828 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.100784    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-podinfo\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.100828 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.100828    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6152dd-f966-4d23-9d91-42ab3743b64a-istiod-ca-cert\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.100979    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhtp\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-kube-api-access-jbhtp\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101119 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101077    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-socket\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101127    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-envoy\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101154    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-data\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101333 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-credential-socket\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101391 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101345    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-certs\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101391 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101376    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-token\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101503 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101472    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-socket\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101563 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101527    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-data\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101563 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101546    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6152dd-f966-4d23-9d91-42ab3743b64a-istiod-ca-cert\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101662 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101578    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-credential-socket\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.101749 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.101727    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-certs\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.103454 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.103431    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-podinfo\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.103611 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.103592    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-envoy\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.110852 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.110812    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhtp\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-kube-api-access-jbhtp\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.111276 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.111257    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-token\") pod \"e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") " pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.204366 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.204265    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:00.482262 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.482231    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"]
Sep 09 10:50:00.483696 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:50:00.483671    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6152dd_f966_4d23_9d91_42ab3743b64a.slice/crio-74e5c02f24de90d091b40f0cfa17300e05841043c08fcb895fa2e65552815902 WatchSource:0}: Error finding container 74e5c02f24de90d091b40f0cfa17300e05841043c08fcb895fa2e65552815902: Status 404 returned error can't find the container with id 74e5c02f24de90d091b40f0cfa17300e05841043c08fcb895fa2e65552815902
Sep 09 10:50:00.486158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.486124    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:00.486261 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.486203    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:00.486261 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.486246    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:00.739968 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.739853    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" event={"ID":"5a6152dd-f966-4d23-9d91-42ab3743b64a","Type":"ContainerStarted","Data":"7f324b6e3093a4387fda3268821315e396c3940a173a7d1411056966089016e7"}
Sep 09 10:50:00.739968 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.739951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" event={"ID":"5a6152dd-f966-4d23-9d91-42ab3743b64a","Type":"ContainerStarted","Data":"74e5c02f24de90d091b40f0cfa17300e05841043c08fcb895fa2e65552815902"}
Sep 09 10:50:00.759406 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:00.759341    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" podStartSLOduration=1.759317394 podStartE2EDuration="1.759317394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:00.758364445 +0000 UTC m=+1442.813923521" watchObservedRunningTime="2026-09-09 10:50:00.759317394 +0000 UTC m=+1442.814876439"
Sep 09 10:50:01.204642 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:01.204605    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:01.210340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:01.210317    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:01.746491 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:01.746456    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:01.747332 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:01.747313    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:11.416841 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:11.416804    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:50:21.888701 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:21.888666    2586 generic.go:356] "Generic (PLEG): container finished" podID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerID="7f324b6e3093a4387fda3268821315e396c3940a173a7d1411056966089016e7" exitCode=0
Sep 09 10:50:21.889137 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:21.888739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" event={"ID":"5a6152dd-f966-4d23-9d91-42ab3743b64a","Type":"ContainerDied","Data":"7f324b6e3093a4387fda3268821315e396c3940a173a7d1411056966089016e7"}
Sep 09 10:50:21.889181 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:21.889145    2586 scope.go:117] "RemoveContainer" containerID="7f324b6e3093a4387fda3268821315e396c3940a173a7d1411056966089016e7"
Sep 09 10:50:21.890173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:21.890145    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:21.890234 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:21.890212    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:21.890270 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:21.890247    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:22.897304 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:22.897262    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" event={"ID":"5a6152dd-f966-4d23-9d91-42ab3743b64a","Type":"ContainerStarted","Data":"33079d934cf2146d8218f6d0d5e986283807f3b1425069c577fffac8e55269f7"}
Sep 09 10:50:23.205348 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:23.205260    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:23.205505 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:23.205433    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:50:23.205548 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:23.205521    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 10:50:24.204862 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:24.204826    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:50:24.205286 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:24.204881    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 10:50:25.204930 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:25.204887    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:50:25.205326 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:25.204957    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 10:50:26.205032 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:26.204994    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:50:26.205444 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:26.205051    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 10:50:27.204929 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:27.204867    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 10:50:27.205107 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:27.204966    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 10:50:28.209567 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.209536    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:28.209951 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.209813    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:28.210531 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.210513    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:50:28.318590 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.318558    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"]
Sep 09 10:50:28.319081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.319066    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy"
Sep 09 10:50:28.319132 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.319083    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="020c1159-38bc-4914-b204-c3d7f1f2ca49" containerName="istio-proxy"
Sep 09 10:50:28.322569 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.322545    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.324875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.324855    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a03d9e-openshift-default-dockercfg-7d6kh\""
Sep 09 10:50:28.339124 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.339098    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"]
Sep 09 10:50:28.487340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487256    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-credential-socket\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.487340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487304    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-podinfo\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.487340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487326    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-socket\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.487628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487398    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-token\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.487628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487427    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-data\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.487628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487466    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-certs\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.487628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487487    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jj5p\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-kube-api-access-6jj5p\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.487628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487563    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istiod-ca-cert\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.487628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.487599    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-envoy\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.588554 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.588513    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-podinfo\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.588554 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.588552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-socket\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.588809 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.588668    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-token\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.588809 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.588717    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-data\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.588959 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.588837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-certs\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.588959 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.588911    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jj5p\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-kube-api-access-6jj5p\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.589095 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.589001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istiod-ca-cert\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.589095 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.589041    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-envoy\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.589205 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.589095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-credential-socket\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.589205 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.589131    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-socket\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.589205 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.589159    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-data\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.589444 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.589421    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-certs\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.589602 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.589574    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-credential-socket\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.589806 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.589785    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istiod-ca-cert\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.591309 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.591289    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-podinfo\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.591411 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.591390    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-envoy\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.597690 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.597664    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jj5p\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-kube-api-access-6jj5p\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.602212 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.602187    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-token\") pod \"e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") " pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.633386 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.633363    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:28.918302 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.918270    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"]
Sep 09 10:50:28.919333 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:50:28.919306    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520d15ce_06b6_4da0_92c7_b6fe3b11cf5e.slice/crio-53c6b9e0dfd2c915936a6d5a926108756f2f99fb602a7e0d7abf40b847ce87ea WatchSource:0}: Error finding container 53c6b9e0dfd2c915936a6d5a926108756f2f99fb602a7e0d7abf40b847ce87ea: Status 404 returned error can't find the container with id 53c6b9e0dfd2c915936a6d5a926108756f2f99fb602a7e0d7abf40b847ce87ea
Sep 09 10:50:28.921537 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.921503    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:28.921645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.921582    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:28.921645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.921627    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:28.934106 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:28.934085    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" event={"ID":"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e","Type":"ContainerStarted","Data":"53c6b9e0dfd2c915936a6d5a926108756f2f99fb602a7e0d7abf40b847ce87ea"}
Sep 09 10:50:29.942367 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:29.942325    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" event={"ID":"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e","Type":"ContainerStarted","Data":"ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223"}
Sep 09 10:50:29.964865 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:29.964817    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" podStartSLOduration=1.96480204 podStartE2EDuration="1.96480204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:29.961929141 +0000 UTC m=+1472.017488192" watchObservedRunningTime="2026-09-09 10:50:29.96480204 +0000 UTC m=+1472.020361089"
Sep 09 10:50:30.634382 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:30.634346    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:30.639521 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:30.639499    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:30.948791 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:30.948709    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:30.949984 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:30.949965    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:33.000398 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.000368    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"]
Sep 09 10:50:33.004183 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.004165    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.006689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.006666    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"kube-root-ca.crt\""
Sep 09 10:50:33.007404 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.007379    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"auth-test-model-535b75-kserve-self-signed-certs\""
Sep 09 10:50:33.007504 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.007403    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"default-dockercfg-4nnfx\""
Sep 09 10:50:33.007504 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.007407    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"openshift-service-ca.crt\""
Sep 09 10:50:33.020195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.020163    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"]
Sep 09 10:50:33.104004 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.103969    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"]
Sep 09 10:50:33.107679 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.107659    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.111028 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.111009    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a03d9e-serving-cert\""
Sep 09 10:50:33.120734 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.120710    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"]
Sep 09 10:50:33.138003 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.137975    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-home\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.138255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.138233    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2cn\" (UniqueName: \"kubernetes.io/projected/762278ff-3d65-494f-bafa-56c0ca1f91a4-kube-api-access-gg2cn\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.138482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.138457    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762278ff-3d65-494f-bafa-56c0ca1f91a4-tls-certs\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.138613 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.138499    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-model-cache\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.138613 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.138581    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-dshm\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.138743 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.138651    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-tmp-dir\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.239727 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.239682    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.239951 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.239783    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-dshm\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.239951 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.239843    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-tmp-dir\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.239951 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.239883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-home\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.239951 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.239927    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2cn\" (UniqueName: \"kubernetes.io/projected/762278ff-3d65-494f-bafa-56c0ca1f91a4-kube-api-access-gg2cn\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.240185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.239994    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.240185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.240023    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnnl\" (UniqueName: \"kubernetes.io/projected/7cef9733-957d-4ee3-84ad-16eac3d4c56c-kube-api-access-7rnnl\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.240185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.240070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762278ff-3d65-494f-bafa-56c0ca1f91a4-tls-certs\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.240185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.240118    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef9733-957d-4ee3-84ad-16eac3d4c56c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.240185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.240157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-model-cache\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.240461 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.240358    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-tmp-dir\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.240461 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.240399    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-home\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.240573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.240502    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-model-cache\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.242080 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.242061    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-dshm\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.242495 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.242476    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762278ff-3d65-494f-bafa-56c0ca1f91a4-tls-certs\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.248194 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.248175    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2cn\" (UniqueName: \"kubernetes.io/projected/762278ff-3d65-494f-bafa-56c0ca1f91a4-kube-api-access-gg2cn\") pod \"auth-test-model-535b75-kserve-694cd746bc-rprmt\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") " pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.316467 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.316381    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:33.341173 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.341146    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.341318 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.341181    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnnl\" (UniqueName: \"kubernetes.io/projected/7cef9733-957d-4ee3-84ad-16eac3d4c56c-kube-api-access-7rnnl\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.341318 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.341212    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef9733-957d-4ee3-84ad-16eac3d4c56c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.341436 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:50:33.341320    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-serving-cert: secret "maas-api-e2e-shared-b-w2-a03d9e-serving-cert" not found
Sep 09 10:50:33.341436 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.341396    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.341718 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:50:33.341583    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls podName:7cef9733-957d-4ee3-84ad-16eac3d4c56c nodeName:}" failed. No retries permitted until 2026-09-09 10:50:33.841561056 +0000 UTC m=+1475.897120088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9" (UID: "7cef9733-957d-4ee3-84ad-16eac3d4c56c") : secret "maas-api-e2e-shared-b-w2-a03d9e-serving-cert" not found
Sep 09 10:50:33.342032 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.342008    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef9733-957d-4ee3-84ad-16eac3d4c56c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.344188 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.344113    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.353488 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.353462    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnnl\" (UniqueName: \"kubernetes.io/projected/7cef9733-957d-4ee3-84ad-16eac3d4c56c-kube-api-access-7rnnl\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.602667 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.602639    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"]
Sep 09 10:50:33.604813 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:50:33.604790    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod762278ff_3d65_494f_bafa_56c0ca1f91a4.slice/crio-86b6baa2c3c2a5f085a4af62205d38c4995f6b49d1cbe39b858df5fabd156192 WatchSource:0}: Error finding container 86b6baa2c3c2a5f085a4af62205d38c4995f6b49d1cbe39b858df5fabd156192: Status 404 returned error can't find the container with id 86b6baa2c3c2a5f085a4af62205d38c4995f6b49d1cbe39b858df5fabd156192
Sep 09 10:50:33.847097 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.847060    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.849457 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.849432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:33.968466 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.968426    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt" event={"ID":"762278ff-3d65-494f-bafa-56c0ca1f91a4","Type":"ContainerStarted","Data":"b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21"}
Sep 09 10:50:33.968466 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.968468    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt" event={"ID":"762278ff-3d65-494f-bafa-56c0ca1f91a4","Type":"ContainerStarted","Data":"86b6baa2c3c2a5f085a4af62205d38c4995f6b49d1cbe39b858df5fabd156192"}
Sep 09 10:50:33.999780 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:33.999720    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt" podStartSLOduration=1.999700486 podStartE2EDuration="1.999700486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:33.998168816 +0000 UTC m=+1476.053727858" watchObservedRunningTime="2026-09-09 10:50:33.999700486 +0000 UTC m=+1476.055259540"
Sep 09 10:50:34.019282 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:34.019248    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:34.298209 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:34.298178    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"]
Sep 09 10:50:34.299285 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:50:34.299261    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cef9733_957d_4ee3_84ad_16eac3d4c56c.slice/crio-1cc5bf7de1a5b28bdf8ee983ec95d85d39f85cc4596d00274cd05047e71d3621 WatchSource:0}: Error finding container 1cc5bf7de1a5b28bdf8ee983ec95d85d39f85cc4596d00274cd05047e71d3621: Status 404 returned error can't find the container with id 1cc5bf7de1a5b28bdf8ee983ec95d85d39f85cc4596d00274cd05047e71d3621
Sep 09 10:50:34.976823 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:34.976724    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9" event={"ID":"7cef9733-957d-4ee3-84ad-16eac3d4c56c","Type":"ContainerStarted","Data":"a4e605fd0e99985c30c4625a8ea9e7990af3409ffee24226bbcda1abb82082b4"}
Sep 09 10:50:34.976823 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:34.976772    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9" event={"ID":"7cef9733-957d-4ee3-84ad-16eac3d4c56c","Type":"ContainerStarted","Data":"1cc5bf7de1a5b28bdf8ee983ec95d85d39f85cc4596d00274cd05047e71d3621"}
Sep 09 10:50:34.995377 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:34.995317    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9" podStartSLOduration=1.628752679 podStartE2EDuration="1.995290988s" totalImagesPullingTime="366.538309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:34.994942937 +0000 UTC m=+1477.050501997" watchObservedRunningTime="2026-09-09 10:50:34.995290988 +0000 UTC m=+1477.050850039"
Sep 09 10:50:35.983729 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:35.983701    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:40.808198 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:40.808151    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:50:41.996038 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:41.996010    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:50:43.316726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:43.316683    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:43.317255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:43.316742    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:43.354662 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:43.354632    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:44.062921 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:44.062875    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:50:52.102389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:52.102352    2586 generic.go:356] "Generic (PLEG): container finished" podID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerID="ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223" exitCode=0
Sep 09 10:50:52.102792 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:52.102428    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" event={"ID":"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e","Type":"ContainerDied","Data":"ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223"}
Sep 09 10:50:52.102866 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:52.102849    2586 scope.go:117] "RemoveContainer" containerID="ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223"
Sep 09 10:50:52.104003 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:52.103974    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:52.104073 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:52.104057    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:52.104113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:52.104095    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:50:53.111114 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:53.111077    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" event={"ID":"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e","Type":"ContainerStarted","Data":"f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f"}
Sep 09 10:50:53.634418 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:53.634372    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:53.634769 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:53.634739    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 09 10:50:53.634870 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:53.634817    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 09 10:50:54.635047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:54.634558    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 09 10:50:54.635047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:54.634627    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 09 10:50:55.634992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:55.634454    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 09 10:50:55.634992 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:55.634521    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 09 10:50:56.634520 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:56.634480    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 09 10:50:56.634826 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:56.634794    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 09 10:50:57.444557 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.444507    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:50:57.550203 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.550167    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:50:57.556151 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.556125    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:50:57.558664 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.558632    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:50:57.558945 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.558929    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dqfgn\""
Sep 09 10:50:57.642478 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.642440    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 09 10:50:57.642764 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.642732    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 09 10:50:57.693117 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.693079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdnjt\" (UniqueName: \"kubernetes.io/projected/7a20ad90-6dbb-45a6-b996-17e21eca08aa-kube-api-access-gdnjt\") pod \"test-curl-7932-139930358200048\" (UID: \"7a20ad90-6dbb-45a6-b996-17e21eca08aa\") " pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:50:57.794491 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.794396    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdnjt\" (UniqueName: \"kubernetes.io/projected/7a20ad90-6dbb-45a6-b996-17e21eca08aa-kube-api-access-gdnjt\") pod \"test-curl-7932-139930358200048\" (UID: \"7a20ad90-6dbb-45a6-b996-17e21eca08aa\") " pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:50:57.804514 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.804485    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdnjt\" (UniqueName: \"kubernetes.io/projected/7a20ad90-6dbb-45a6-b996-17e21eca08aa-kube-api-access-gdnjt\") pod \"test-curl-7932-139930358200048\" (UID: \"7a20ad90-6dbb-45a6-b996-17e21eca08aa\") " pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:50:57.867284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:57.867249    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:50:58.142324 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:58.140550    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:50:58.152923 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:50:58.148006    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a20ad90_6dbb_45a6_b996_17e21eca08aa.slice/crio-35cabbf38b16f2848344b147b90b36e383d7cbaa582d0166c0b28c18e985c249 WatchSource:0}: Error finding container 35cabbf38b16f2848344b147b90b36e383d7cbaa582d0166c0b28c18e985c249: Status 404 returned error can't find the container with id 35cabbf38b16f2848344b147b90b36e383d7cbaa582d0166c0b28c18e985c249
Sep 09 10:50:58.604739 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:58.604653    2586 scope.go:117] "RemoveContainer" containerID="3d20621394bc85c445155962b5b90499d72f96de4e414c0794a3f7d0a12b6cd0"
Sep 09 10:50:58.637564 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:58.637538    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:58.637809 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:58.637793    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:58.638521 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:58.638502    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:50:59.163099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:59.163017    2586 generic.go:356] "Generic (PLEG): container finished" podID="7a20ad90-6dbb-45a6-b996-17e21eca08aa" containerID="cdaee3f4c2949f2faf5be5d46d951d7687fedd11be944be2e96bfc3ee13e32b4" exitCode=0
Sep 09 10:50:59.163099 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:59.163056    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7932-139930358200048" event={"ID":"7a20ad90-6dbb-45a6-b996-17e21eca08aa","Type":"ContainerDied","Data":"cdaee3f4c2949f2faf5be5d46d951d7687fedd11be944be2e96bfc3ee13e32b4"}
Sep 09 10:50:59.163296 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:50:59.163101    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7932-139930358200048" event={"ID":"7a20ad90-6dbb-45a6-b996-17e21eca08aa","Type":"ContainerStarted","Data":"35cabbf38b16f2848344b147b90b36e383d7cbaa582d0166c0b28c18e985c249"}
Sep 09 10:51:00.298584 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.298562    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:00.336761 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.336728    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7932-139930358200048_7a20ad90-6dbb-45a6-b996-17e21eca08aa/test-curl-7932-139930358200048/0.log"
Sep 09 10:51:00.361350 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.361323    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:51:00.366000 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.365976    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:51:00.421128 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.421058    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdnjt\" (UniqueName: \"kubernetes.io/projected/7a20ad90-6dbb-45a6-b996-17e21eca08aa-kube-api-access-gdnjt\") pod \"7a20ad90-6dbb-45a6-b996-17e21eca08aa\" (UID: \"7a20ad90-6dbb-45a6-b996-17e21eca08aa\") "
Sep 09 10:51:00.423129 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.423100    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a20ad90-6dbb-45a6-b996-17e21eca08aa-kube-api-access-gdnjt" (OuterVolumeSpecName: "kube-api-access-gdnjt") pod "7a20ad90-6dbb-45a6-b996-17e21eca08aa" (UID: "7a20ad90-6dbb-45a6-b996-17e21eca08aa"). InnerVolumeSpecName "kube-api-access-gdnjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:51:00.475159 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.475130    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a20ad90-6dbb-45a6-b996-17e21eca08aa" path="/var/lib/kubelet/pods/7a20ad90-6dbb-45a6-b996-17e21eca08aa/volumes"
Sep 09 10:51:00.491813 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.491792    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:51:00.492255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.492240    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a20ad90-6dbb-45a6-b996-17e21eca08aa" containerName="test-curl-7932-139930358200048"
Sep 09 10:51:00.492310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.492256    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a20ad90-6dbb-45a6-b996-17e21eca08aa" containerName="test-curl-7932-139930358200048"
Sep 09 10:51:00.492355 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.492322    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a20ad90-6dbb-45a6-b996-17e21eca08aa" containerName="test-curl-7932-139930358200048"
Sep 09 10:51:00.495441 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.495425    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:00.501497 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.501470    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:51:00.522140 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.522116    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdnjt\" (UniqueName: \"kubernetes.io/projected/7a20ad90-6dbb-45a6-b996-17e21eca08aa-kube-api-access-gdnjt\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:51:00.623586 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.623534    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzsbh\" (UniqueName: \"kubernetes.io/projected/8e6d9464-593f-4986-8a10-234983ae266e-kube-api-access-vzsbh\") pod \"test-curl-7932-139930358200048\" (UID: \"8e6d9464-593f-4986-8a10-234983ae266e\") " pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:00.724758 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.724677    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzsbh\" (UniqueName: \"kubernetes.io/projected/8e6d9464-593f-4986-8a10-234983ae266e-kube-api-access-vzsbh\") pod \"test-curl-7932-139930358200048\" (UID: \"8e6d9464-593f-4986-8a10-234983ae266e\") " pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:00.733183 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.733160    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzsbh\" (UniqueName: \"kubernetes.io/projected/8e6d9464-593f-4986-8a10-234983ae266e-kube-api-access-vzsbh\") pod \"test-curl-7932-139930358200048\" (UID: \"8e6d9464-593f-4986-8a10-234983ae266e\") " pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:00.806357 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:00.806324    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:01.075421 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:01.075395    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:51:01.076586 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:51:01.076558    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e6d9464_593f_4986_8a10_234983ae266e.slice/crio-0257afc423deb9afe6118c0ecc56c6ac5f133334dcdccc4752046926ecd2bc54 WatchSource:0}: Error finding container 0257afc423deb9afe6118c0ecc56c6ac5f133334dcdccc4752046926ecd2bc54: Status 404 returned error can't find the container with id 0257afc423deb9afe6118c0ecc56c6ac5f133334dcdccc4752046926ecd2bc54
Sep 09 10:51:01.176185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:01.176157    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:01.176386 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:01.176196    2586 scope.go:117] "RemoveContainer" containerID="cdaee3f4c2949f2faf5be5d46d951d7687fedd11be944be2e96bfc3ee13e32b4"
Sep 09 10:51:01.177701 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:01.177669    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7932-139930358200048" event={"ID":"8e6d9464-593f-4986-8a10-234983ae266e","Type":"ContainerStarted","Data":"0257afc423deb9afe6118c0ecc56c6ac5f133334dcdccc4752046926ecd2bc54"}
Sep 09 10:51:02.186116 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:02.186076    2586 generic.go:356] "Generic (PLEG): container finished" podID="8e6d9464-593f-4986-8a10-234983ae266e" containerID="1676c8fbd5ecf57fe802b3413910c4522954935a471c5f21c46ba05b489255a8" exitCode=0
Sep 09 10:51:02.186526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:02.186159    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7932-139930358200048" event={"ID":"8e6d9464-593f-4986-8a10-234983ae266e","Type":"ContainerDied","Data":"1676c8fbd5ecf57fe802b3413910c4522954935a471c5f21c46ba05b489255a8"}
Sep 09 10:51:03.330024 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.329999    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:03.351081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.351054    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7932-139930358200048_8e6d9464-593f-4986-8a10-234983ae266e/test-curl-7932-139930358200048/0.log"
Sep 09 10:51:03.375714 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.375673    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:51:03.379592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.379569    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7932-139930358200048"]
Sep 09 10:51:03.450997 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.450918    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzsbh\" (UniqueName: \"kubernetes.io/projected/8e6d9464-593f-4986-8a10-234983ae266e-kube-api-access-vzsbh\") pod \"8e6d9464-593f-4986-8a10-234983ae266e\" (UID: \"8e6d9464-593f-4986-8a10-234983ae266e\") "
Sep 09 10:51:03.453031 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.453005    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6d9464-593f-4986-8a10-234983ae266e-kube-api-access-vzsbh" (OuterVolumeSpecName: "kube-api-access-vzsbh") pod "8e6d9464-593f-4986-8a10-234983ae266e" (UID: "8e6d9464-593f-4986-8a10-234983ae266e"). InnerVolumeSpecName "kube-api-access-vzsbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:51:03.512047 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.512020    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:03.512431 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.512417    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6d9464-593f-4986-8a10-234983ae266e" containerName="test-curl-7932-139930358200048"
Sep 09 10:51:03.512480 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.512433    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6d9464-593f-4986-8a10-234983ae266e" containerName="test-curl-7932-139930358200048"
Sep 09 10:51:03.512527 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.512517    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6d9464-593f-4986-8a10-234983ae266e" containerName="test-curl-7932-139930358200048"
Sep 09 10:51:03.515638 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.515621    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:03.524374 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.523908    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:03.551847 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.551812    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzsbh\" (UniqueName: \"kubernetes.io/projected/8e6d9464-593f-4986-8a10-234983ae266e-kube-api-access-vzsbh\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:51:03.652577 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.652532    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn84\" (UniqueName: \"kubernetes.io/projected/8a1103b2-4bb0-4c7c-8e35-fb961e81ff13-kube-api-access-sfn84\") pod \"test-curl-7932-139930358200624\" (UID: \"8a1103b2-4bb0-4c7c-8e35-fb961e81ff13\") " pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:03.754049 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.753957    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn84\" (UniqueName: \"kubernetes.io/projected/8a1103b2-4bb0-4c7c-8e35-fb961e81ff13-kube-api-access-sfn84\") pod \"test-curl-7932-139930358200624\" (UID: \"8a1103b2-4bb0-4c7c-8e35-fb961e81ff13\") " pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:03.762580 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.762546    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn84\" (UniqueName: \"kubernetes.io/projected/8a1103b2-4bb0-4c7c-8e35-fb961e81ff13-kube-api-access-sfn84\") pod \"test-curl-7932-139930358200624\" (UID: \"8a1103b2-4bb0-4c7c-8e35-fb961e81ff13\") " pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:03.826968 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:03.826934    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:04.097723 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:04.097696    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:04.199212 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:04.199182    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:04.199382 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:04.199191    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0257afc423deb9afe6118c0ecc56c6ac5f133334dcdccc4752046926ecd2bc54"
Sep 09 10:51:04.200574 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:04.200541    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7932-139930358200624" event={"ID":"8a1103b2-4bb0-4c7c-8e35-fb961e81ff13","Type":"ContainerStarted","Data":"f5e52bd346ba049a4a79c929fde825b6a133bd955c89fff8012352b3ee78f1a1"}
Sep 09 10:51:04.209804 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:51:04.209778    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7932-139930358200048\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="8e6d9464-593f-4986-8a10-234983ae266e" pod="opendatahub/test-curl-7932-139930358200048"
Sep 09 10:51:04.475673 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:04.475639    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6d9464-593f-4986-8a10-234983ae266e" path="/var/lib/kubelet/pods/8e6d9464-593f-4986-8a10-234983ae266e/volumes"
Sep 09 10:51:05.210168 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:05.210142    2586 generic.go:356] "Generic (PLEG): container finished" podID="8a1103b2-4bb0-4c7c-8e35-fb961e81ff13" containerID="cafc82df60a89df5ed402135efdc923539d08dba5298990062797d95a12eaa09" exitCode=0
Sep 09 10:51:05.210284 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:05.210224    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7932-139930358200624" event={"ID":"8a1103b2-4bb0-4c7c-8e35-fb961e81ff13","Type":"ContainerDied","Data":"cafc82df60a89df5ed402135efdc923539d08dba5298990062797d95a12eaa09"}
Sep 09 10:51:06.358359 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.358328    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:06.379067 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.379041    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7932-139930358200624_8a1103b2-4bb0-4c7c-8e35-fb961e81ff13/test-curl-7932-139930358200624/0.log"
Sep 09 10:51:06.402567 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.402532    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:06.405743 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.405718    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:06.484853 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.484767    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfn84\" (UniqueName: \"kubernetes.io/projected/8a1103b2-4bb0-4c7c-8e35-fb961e81ff13-kube-api-access-sfn84\") pod \"8a1103b2-4bb0-4c7c-8e35-fb961e81ff13\" (UID: \"8a1103b2-4bb0-4c7c-8e35-fb961e81ff13\") "
Sep 09 10:51:06.486945 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.486924    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1103b2-4bb0-4c7c-8e35-fb961e81ff13-kube-api-access-sfn84" (OuterVolumeSpecName: "kube-api-access-sfn84") pod "8a1103b2-4bb0-4c7c-8e35-fb961e81ff13" (UID: "8a1103b2-4bb0-4c7c-8e35-fb961e81ff13"). InnerVolumeSpecName "kube-api-access-sfn84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:51:06.524570 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.524544    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:06.525000 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.524985    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1103b2-4bb0-4c7c-8e35-fb961e81ff13" containerName="test-curl-7932-139930358200624"
Sep 09 10:51:06.525056 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.525001    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1103b2-4bb0-4c7c-8e35-fb961e81ff13" containerName="test-curl-7932-139930358200624"
Sep 09 10:51:06.525101 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.525066    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1103b2-4bb0-4c7c-8e35-fb961e81ff13" containerName="test-curl-7932-139930358200624"
Sep 09 10:51:06.528161 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.528140    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:06.536937 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.536913    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:06.585774 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.585742    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfn84\" (UniqueName: \"kubernetes.io/projected/8a1103b2-4bb0-4c7c-8e35-fb961e81ff13-kube-api-access-sfn84\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:51:06.687261 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.687209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx7t8\" (UniqueName: \"kubernetes.io/projected/30c1b5ba-1b44-42ba-be5f-b014130d1703-kube-api-access-sx7t8\") pod \"test-curl-7932-139930358200624\" (UID: \"30c1b5ba-1b44-42ba-be5f-b014130d1703\") " pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:06.788699 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.788592    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx7t8\" (UniqueName: \"kubernetes.io/projected/30c1b5ba-1b44-42ba-be5f-b014130d1703-kube-api-access-sx7t8\") pod \"test-curl-7932-139930358200624\" (UID: \"30c1b5ba-1b44-42ba-be5f-b014130d1703\") " pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:06.796922 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.796879    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx7t8\" (UniqueName: \"kubernetes.io/projected/30c1b5ba-1b44-42ba-be5f-b014130d1703-kube-api-access-sx7t8\") pod \"test-curl-7932-139930358200624\" (UID: \"30c1b5ba-1b44-42ba-be5f-b014130d1703\") " pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:06.838987 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:06.838965    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:07.111737 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:07.111704    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:07.112633 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:51:07.112603    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30c1b5ba_1b44_42ba_be5f_b014130d1703.slice/crio-0181d1be829d48b84e6a68ec3be1764aa46fcaf3ef0d86011ddec81d7107f2a5 WatchSource:0}: Error finding container 0181d1be829d48b84e6a68ec3be1764aa46fcaf3ef0d86011ddec81d7107f2a5: Status 404 returned error can't find the container with id 0181d1be829d48b84e6a68ec3be1764aa46fcaf3ef0d86011ddec81d7107f2a5
Sep 09 10:51:07.224609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:07.224578    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:07.224760 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:07.224585    2586 scope.go:117] "RemoveContainer" containerID="cafc82df60a89df5ed402135efdc923539d08dba5298990062797d95a12eaa09"
Sep 09 10:51:07.226114 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:07.225845    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7932-139930358200624" event={"ID":"30c1b5ba-1b44-42ba-be5f-b014130d1703","Type":"ContainerStarted","Data":"0181d1be829d48b84e6a68ec3be1764aa46fcaf3ef0d86011ddec81d7107f2a5"}
Sep 09 10:51:08.234685 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:08.234650    2586 generic.go:356] "Generic (PLEG): container finished" podID="30c1b5ba-1b44-42ba-be5f-b014130d1703" containerID="c04e1c9c03ed658cbbe00c56b6d8a3be3ee1ac44b811001be07e14f5a92c2fce" exitCode=0
Sep 09 10:51:08.235135 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:08.234728    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7932-139930358200624" event={"ID":"30c1b5ba-1b44-42ba-be5f-b014130d1703","Type":"ContainerDied","Data":"c04e1c9c03ed658cbbe00c56b6d8a3be3ee1ac44b811001be07e14f5a92c2fce"}
Sep 09 10:51:08.475863 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:08.475827    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a1103b2-4bb0-4c7c-8e35-fb961e81ff13" path="/var/lib/kubelet/pods/8a1103b2-4bb0-4c7c-8e35-fb961e81ff13/volumes"
Sep 09 10:51:09.382469 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:09.382444    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:09.401336 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:09.401313    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7932-139930358200624_30c1b5ba-1b44-42ba-be5f-b014130d1703/test-curl-7932-139930358200624/0.log"
Sep 09 10:51:09.415240 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:09.415209    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx7t8\" (UniqueName: \"kubernetes.io/projected/30c1b5ba-1b44-42ba-be5f-b014130d1703-kube-api-access-sx7t8\") pod \"30c1b5ba-1b44-42ba-be5f-b014130d1703\" (UID: \"30c1b5ba-1b44-42ba-be5f-b014130d1703\") "
Sep 09 10:51:09.417220 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:09.417196    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1b5ba-1b44-42ba-be5f-b014130d1703-kube-api-access-sx7t8" (OuterVolumeSpecName: "kube-api-access-sx7t8") pod "30c1b5ba-1b44-42ba-be5f-b014130d1703" (UID: "30c1b5ba-1b44-42ba-be5f-b014130d1703"). InnerVolumeSpecName "kube-api-access-sx7t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:51:09.421525 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:09.421500    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:09.425245 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:09.425225    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7932-139930358200624"]
Sep 09 10:51:09.516713 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:09.516635    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx7t8\" (UniqueName: \"kubernetes.io/projected/30c1b5ba-1b44-42ba-be5f-b014130d1703-kube-api-access-sx7t8\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:51:10.101538 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:10.101499    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:51:10.247726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:10.247698    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0181d1be829d48b84e6a68ec3be1764aa46fcaf3ef0d86011ddec81d7107f2a5"
Sep 09 10:51:10.247726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:10.247725    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:10.257516 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:51:10.257479    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7932-139930358200624\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="30c1b5ba-1b44-42ba-be5f-b014130d1703" pod="opendatahub/test-curl-7932-139930358200624"
Sep 09 10:51:10.482537 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:10.482500    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c1b5ba-1b44-42ba-be5f-b014130d1703" path="/var/lib/kubelet/pods/30c1b5ba-1b44-42ba-be5f-b014130d1703/volumes"
Sep 09 10:51:31.105570 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:31.105484    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:51:45.603375 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:45.603342    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:51:58.626804 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:58.626768    2586 scope.go:117] "RemoveContainer" containerID="6eebef8f7b0eeea39c94282e67cfff91bbfefaf78a8c92ef49f598f62808d7e5"
Sep 09 10:51:58.636032 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:51:58.636013    2586 scope.go:117] "RemoveContainer" containerID="38ecbb6c717c2afb2f61f1b1d965bf052ac0d257ff1b42c9bac317bf528a4327"
Sep 09 10:52:03.004064 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:03.004026    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:52:04.043036 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.042995    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"]
Sep 09 10:52:04.043486 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.043452    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1b5ba-1b44-42ba-be5f-b014130d1703" containerName="test-curl-7932-139930358200624"
Sep 09 10:52:04.043486 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.043463    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1b5ba-1b44-42ba-be5f-b014130d1703" containerName="test-curl-7932-139930358200624"
Sep 09 10:52:04.043596 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.043538    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c1b5ba-1b44-42ba-be5f-b014130d1703" containerName="test-curl-7932-139930358200624"
Sep 09 10:52:04.046241 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.046223    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.051013 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.050992    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bbfd5f59\"/\"openshift-service-ca.crt\""
Sep 09 10:52:04.051181 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.051020    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bbfd5f59\"/\"default-dockercfg-6fmc8\""
Sep 09 10:52:04.051248 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.051234    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bbfd5f59\"/\"kube-root-ca.crt\""
Sep 09 10:52:04.051289 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.051259    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bbfd5f59\"/\"e2e-lifecycle-model-bbfd5f59-kserve-self-signed-certs\""
Sep 09 10:52:04.060214 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.060192    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"]
Sep 09 10:52:04.153814 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.153783    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tls-certs\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.154016 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.153832    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-model-cache\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.154016 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.153870    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tmp-dir\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.154016 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.153942    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-dshm\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.154016 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.153976    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzfjf\" (UniqueName: \"kubernetes.io/projected/b7759d4c-1703-4dd0-9a75-d6eb260cc964-kube-api-access-qzfjf\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.154016 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.153999    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-home\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.255353 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.255315    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tls-certs\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.255544 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.255371    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-model-cache\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.255544 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.255529    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tmp-dir\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.255675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.255590    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-dshm\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.255675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.255639    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzfjf\" (UniqueName: \"kubernetes.io/projected/b7759d4c-1703-4dd0-9a75-d6eb260cc964-kube-api-access-qzfjf\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.255675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.255669    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-home\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.255826 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.255761    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-model-cache\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.255902 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.255874    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tmp-dir\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.256036 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.256011    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-home\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.257765 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.257746    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-dshm\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.257969 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.257945    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tls-certs\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.263983 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.263963    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzfjf\" (UniqueName: \"kubernetes.io/projected/b7759d4c-1703-4dd0-9a75-d6eb260cc964-kube-api-access-qzfjf\") pod \"e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") " pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.356394 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.356296    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:04.631734 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:04.631706    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"]
Sep 09 10:52:04.632297 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:52:04.632271    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7759d4c_1703_4dd0_9a75_d6eb260cc964.slice/crio-cebbcd53a9da53a0fdf3e4240ed6f92fdfe37e0c28d42a8946ffd9d06eb626b1 WatchSource:0}: Error finding container cebbcd53a9da53a0fdf3e4240ed6f92fdfe37e0c28d42a8946ffd9d06eb626b1: Status 404 returned error can't find the container with id cebbcd53a9da53a0fdf3e4240ed6f92fdfe37e0c28d42a8946ffd9d06eb626b1
Sep 09 10:52:05.605391 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:05.605359    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252" event={"ID":"b7759d4c-1703-4dd0-9a75-d6eb260cc964","Type":"ContainerStarted","Data":"2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337"}
Sep 09 10:52:05.605843 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:05.605401    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252" event={"ID":"b7759d4c-1703-4dd0-9a75-d6eb260cc964","Type":"ContainerStarted","Data":"cebbcd53a9da53a0fdf3e4240ed6f92fdfe37e0c28d42a8946ffd9d06eb626b1"}
Sep 09 10:52:05.625988 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:05.625941    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252" podStartSLOduration=1.6259256290000001 podStartE2EDuration="1.625925629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:52:05.622871621 +0000 UTC m=+1567.678430673" watchObservedRunningTime="2026-09-09 10:52:05.625925629 +0000 UTC m=+1567.681484674"
Sep 09 10:52:14.357096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:14.357052    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:14.357096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:14.357097    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:14.369852 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:14.369826    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:14.673691 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:14.673663    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:52:21.601020 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:21.600981    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:52:33.732140 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:33.732103    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7935-65f57e"]
Sep 09 10:52:33.735866 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:33.735843    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:33.738096 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:33.738075    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dqfgn\""
Sep 09 10:52:33.740832 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:33.740809    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7935-65f57e"]
Sep 09 10:52:33.854091 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:33.854052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm44v\" (UniqueName: \"kubernetes.io/projected/5bc05b8b-81a9-48bc-a8c3-a6daf3427839-kube-api-access-dm44v\") pod \"mt-curl-7935-65f57e\" (UID: \"5bc05b8b-81a9-48bc-a8c3-a6daf3427839\") " pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:33.955279 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:33.955242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm44v\" (UniqueName: \"kubernetes.io/projected/5bc05b8b-81a9-48bc-a8c3-a6daf3427839-kube-api-access-dm44v\") pod \"mt-curl-7935-65f57e\" (UID: \"5bc05b8b-81a9-48bc-a8c3-a6daf3427839\") " pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:33.964066 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:33.964032    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm44v\" (UniqueName: \"kubernetes.io/projected/5bc05b8b-81a9-48bc-a8c3-a6daf3427839-kube-api-access-dm44v\") pod \"mt-curl-7935-65f57e\" (UID: \"5bc05b8b-81a9-48bc-a8c3-a6daf3427839\") " pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:34.047255 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:34.047180    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:34.320466 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:34.320437    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7935-65f57e"]
Sep 09 10:52:34.321802 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:52:34.321776    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bc05b8b_81a9_48bc_a8c3_a6daf3427839.slice/crio-d52de2aee6b768b90977776ebf9877eaced535ad66f63fb25197c979d0bcbf32 WatchSource:0}: Error finding container d52de2aee6b768b90977776ebf9877eaced535ad66f63fb25197c979d0bcbf32: Status 404 returned error can't find the container with id d52de2aee6b768b90977776ebf9877eaced535ad66f63fb25197c979d0bcbf32
Sep 09 10:52:34.789822 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:34.789793    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7935-65f57e" event={"ID":"5bc05b8b-81a9-48bc-a8c3-a6daf3427839","Type":"ContainerStarted","Data":"d52de2aee6b768b90977776ebf9877eaced535ad66f63fb25197c979d0bcbf32"}
Sep 09 10:52:35.797317 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:35.797276    2586 generic.go:356] "Generic (PLEG): container finished" podID="5bc05b8b-81a9-48bc-a8c3-a6daf3427839" containerID="693f4d9110f53fbbdb97b6298e5d039dfd66edddd1dbc493b87903dacffdbd00" exitCode=0
Sep 09 10:52:35.797698 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:35.797358    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7935-65f57e" event={"ID":"5bc05b8b-81a9-48bc-a8c3-a6daf3427839","Type":"ContainerDied","Data":"693f4d9110f53fbbdb97b6298e5d039dfd66edddd1dbc493b87903dacffdbd00"}
Sep 09 10:52:36.937790 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:36.937760    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:36.959140 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:36.959111    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7935-65f57e_5bc05b8b-81a9-48bc-a8c3-a6daf3427839/mt-curl-7935-65f57e/0.log"
Sep 09 10:52:36.983235 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:36.983203    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm44v\" (UniqueName: \"kubernetes.io/projected/5bc05b8b-81a9-48bc-a8c3-a6daf3427839-kube-api-access-dm44v\") pod \"5bc05b8b-81a9-48bc-a8c3-a6daf3427839\" (UID: \"5bc05b8b-81a9-48bc-a8c3-a6daf3427839\") "
Sep 09 10:52:36.983373 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:36.983261    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7935-65f57e"]
Sep 09 10:52:36.985410 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:36.985369    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc05b8b-81a9-48bc-a8c3-a6daf3427839-kube-api-access-dm44v" (OuterVolumeSpecName: "kube-api-access-dm44v") pod "5bc05b8b-81a9-48bc-a8c3-a6daf3427839" (UID: "5bc05b8b-81a9-48bc-a8c3-a6daf3427839"). InnerVolumeSpecName "kube-api-access-dm44v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:52:36.987703 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:36.987681    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7935-65f57e"]
Sep 09 10:52:37.084937 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:37.084829    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm44v\" (UniqueName: \"kubernetes.io/projected/5bc05b8b-81a9-48bc-a8c3-a6daf3427839-kube-api-access-dm44v\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:52:37.811524 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:37.811495    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d52de2aee6b768b90977776ebf9877eaced535ad66f63fb25197c979d0bcbf32"
Sep 09 10:52:37.811717 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:37.811529    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:37.821028 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:52:37.821002    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7935-65f57e\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="5bc05b8b-81a9-48bc-a8c3-a6daf3427839" pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:38.476233 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:52:38.476192    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7935-65f57e\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="5bc05b8b-81a9-48bc-a8c3-a6daf3427839" pod="opendatahub/mt-curl-7935-65f57e"
Sep 09 10:52:38.476233 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:38.476210    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bc05b8b-81a9-48bc-a8c3-a6daf3427839" path="/var/lib/kubelet/pods/5bc05b8b-81a9-48bc-a8c3-a6daf3427839/volumes"
Sep 09 10:52:39.517728 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:39.517690    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"]
Sep 09 10:52:39.620738 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:39.620700    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:52:53.613620 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:53.613534    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:52:54.538632 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.538587    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt" podUID="762278ff-3d65-494f-bafa-56c0ca1f91a4" containerName="main" containerID="cri-o://b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21" gracePeriod=45
Sep 09 10:52:54.791019 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.790959    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:52:54.923327 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.923294    2586 generic.go:356] "Generic (PLEG): container finished" podID="762278ff-3d65-494f-bafa-56c0ca1f91a4" containerID="b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21" exitCode=0
Sep 09 10:52:54.923500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.923344    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"
Sep 09 10:52:54.923500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.923378    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt" event={"ID":"762278ff-3d65-494f-bafa-56c0ca1f91a4","Type":"ContainerDied","Data":"b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21"}
Sep 09 10:52:54.923500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.923420    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt" event={"ID":"762278ff-3d65-494f-bafa-56c0ca1f91a4","Type":"ContainerDied","Data":"86b6baa2c3c2a5f085a4af62205d38c4995f6b49d1cbe39b858df5fabd156192"}
Sep 09 10:52:54.923500 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.923439    2586 scope.go:117] "RemoveContainer" containerID="b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21"
Sep 09 10:52:54.932858 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.932843    2586 scope.go:117] "RemoveContainer" containerID="b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21"
Sep 09 10:52:54.933139 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:52:54.933122    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21\": container with ID starting with b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21 not found: ID does not exist" containerID="b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21"
Sep 09 10:52:54.933228 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.933146    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21"} err="failed to get container status \"b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21\": rpc error: code = NotFound desc = could not find container \"b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21\": container with ID starting with b56ad40c78834e8910c63624c73c0ea7413650f508d04286d516e6ce390eea21 not found: ID does not exist"
Sep 09 10:52:54.965936 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.965882    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-home\") pod \"762278ff-3d65-494f-bafa-56c0ca1f91a4\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") "
Sep 09 10:52:54.966052 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966022    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg2cn\" (UniqueName: \"kubernetes.io/projected/762278ff-3d65-494f-bafa-56c0ca1f91a4-kube-api-access-gg2cn\") pod \"762278ff-3d65-494f-bafa-56c0ca1f91a4\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") "
Sep 09 10:52:54.966094 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966054    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762278ff-3d65-494f-bafa-56c0ca1f91a4-tls-certs\") pod \"762278ff-3d65-494f-bafa-56c0ca1f91a4\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") "
Sep 09 10:52:54.966094 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966080    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-tmp-dir\") pod \"762278ff-3d65-494f-bafa-56c0ca1f91a4\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") "
Sep 09 10:52:54.966202 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966102    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-dshm\") pod \"762278ff-3d65-494f-bafa-56c0ca1f91a4\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") "
Sep 09 10:52:54.966202 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966114    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-home" (OuterVolumeSpecName: "home") pod "762278ff-3d65-494f-bafa-56c0ca1f91a4" (UID: "762278ff-3d65-494f-bafa-56c0ca1f91a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:52:54.966202 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966157    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-model-cache\") pod \"762278ff-3d65-494f-bafa-56c0ca1f91a4\" (UID: \"762278ff-3d65-494f-bafa-56c0ca1f91a4\") "
Sep 09 10:52:54.966398 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966367    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "762278ff-3d65-494f-bafa-56c0ca1f91a4" (UID: "762278ff-3d65-494f-bafa-56c0ca1f91a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:52:54.966492 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966468    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "762278ff-3d65-494f-bafa-56c0ca1f91a4" (UID: "762278ff-3d65-494f-bafa-56c0ca1f91a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:52:54.966567 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966551    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:52:54.966608 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966568    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:52:54.966608 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.966577    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:52:54.968332 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.968310    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762278ff-3d65-494f-bafa-56c0ca1f91a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "762278ff-3d65-494f-bafa-56c0ca1f91a4" (UID: "762278ff-3d65-494f-bafa-56c0ca1f91a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:52:54.968589 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.968571    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-dshm" (OuterVolumeSpecName: "dshm") pod "762278ff-3d65-494f-bafa-56c0ca1f91a4" (UID: "762278ff-3d65-494f-bafa-56c0ca1f91a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:52:54.968651 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:54.968623    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762278ff-3d65-494f-bafa-56c0ca1f91a4-kube-api-access-gg2cn" (OuterVolumeSpecName: "kube-api-access-gg2cn") pod "762278ff-3d65-494f-bafa-56c0ca1f91a4" (UID: "762278ff-3d65-494f-bafa-56c0ca1f91a4"). InnerVolumeSpecName "kube-api-access-gg2cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:52:55.068190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:55.068105    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg2cn\" (UniqueName: \"kubernetes.io/projected/762278ff-3d65-494f-bafa-56c0ca1f91a4-kube-api-access-gg2cn\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:52:55.068190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:55.068135    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762278ff-3d65-494f-bafa-56c0ca1f91a4-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:52:55.068190 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:55.068145    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762278ff-3d65-494f-bafa-56c0ca1f91a4-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:52:55.257349 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:55.257318    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"]
Sep 09 10:52:55.263977 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:55.263942    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/auth-test-model-535b75-kserve-694cd746bc-rprmt"]
Sep 09 10:52:56.475777 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:56.475740    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="762278ff-3d65-494f-bafa-56c0ca1f91a4" path="/var/lib/kubelet/pods/762278ff-3d65-494f-bafa-56c0ca1f91a4/volumes"
Sep 09 10:52:58.673137 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:52:58.673101    2586 scope.go:117] "RemoveContainer" containerID="00ca41c3415113213fb0aca6f40eb580e96c58d83bd83f9b268b89e1472f19a6"
Sep 09 10:53:02.718716 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:02.718680    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"]
Sep 09 10:53:09.709703 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:09.709669    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:53:17.738552 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:17.738507    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252" podUID="b7759d4c-1703-4dd0-9a75-d6eb260cc964" containerName="main" containerID="cri-o://2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337" gracePeriod=45
Sep 09 10:53:17.993630 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:17.993570    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:53:18.060751 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.060709    2586 generic.go:356] "Generic (PLEG): container finished" podID="b7759d4c-1703-4dd0-9a75-d6eb260cc964" containerID="2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337" exitCode=0
Sep 09 10:53:18.060751 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.060757    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"
Sep 09 10:53:18.061030 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.060783    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252" event={"ID":"b7759d4c-1703-4dd0-9a75-d6eb260cc964","Type":"ContainerDied","Data":"2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337"}
Sep 09 10:53:18.061030 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.060827    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252" event={"ID":"b7759d4c-1703-4dd0-9a75-d6eb260cc964","Type":"ContainerDied","Data":"cebbcd53a9da53a0fdf3e4240ed6f92fdfe37e0c28d42a8946ffd9d06eb626b1"}
Sep 09 10:53:18.061030 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.060844    2586 scope.go:117] "RemoveContainer" containerID="2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337"
Sep 09 10:53:18.070910 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.070862    2586 scope.go:117] "RemoveContainer" containerID="2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337"
Sep 09 10:53:18.071243 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:53:18.071223    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337\": container with ID starting with 2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337 not found: ID does not exist" containerID="2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337"
Sep 09 10:53:18.071317 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.071251    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337"} err="failed to get container status \"2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337\": rpc error: code = NotFound desc = could not find container \"2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337\": container with ID starting with 2f738c13c905c463fe64be2e2b4f6056630b8cffa3886d169ef15153e778e337 not found: ID does not exist"
Sep 09 10:53:18.103597 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.103565    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-home\") pod \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") "
Sep 09 10:53:18.103597 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.103602    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tls-certs\") pod \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") "
Sep 09 10:53:18.103829 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.103620    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-model-cache\") pod \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") "
Sep 09 10:53:18.103829 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.103757    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-dshm\") pod \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") "
Sep 09 10:53:18.103829 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.103815    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tmp-dir\") pod \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") "
Sep 09 10:53:18.104021 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.103860    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-home" (OuterVolumeSpecName: "home") pod "b7759d4c-1703-4dd0-9a75-d6eb260cc964" (UID: "b7759d4c-1703-4dd0-9a75-d6eb260cc964"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:18.104021 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.103907    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzfjf\" (UniqueName: \"kubernetes.io/projected/b7759d4c-1703-4dd0-9a75-d6eb260cc964-kube-api-access-qzfjf\") pod \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\" (UID: \"b7759d4c-1703-4dd0-9a75-d6eb260cc964\") "
Sep 09 10:53:18.104021 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.103973    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7759d4c-1703-4dd0-9a75-d6eb260cc964" (UID: "b7759d4c-1703-4dd0-9a75-d6eb260cc964"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:18.104159 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.104104    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7759d4c-1703-4dd0-9a75-d6eb260cc964" (UID: "b7759d4c-1703-4dd0-9a75-d6eb260cc964"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:18.104287 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.104273    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:53:18.104335 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.104290    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:53:18.104335 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.104300    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:53:18.105866 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.105837    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7759d4c-1703-4dd0-9a75-d6eb260cc964" (UID: "b7759d4c-1703-4dd0-9a75-d6eb260cc964"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:53:18.105988 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.105883    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-dshm" (OuterVolumeSpecName: "dshm") pod "b7759d4c-1703-4dd0-9a75-d6eb260cc964" (UID: "b7759d4c-1703-4dd0-9a75-d6eb260cc964"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:53:18.105988 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.105916    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7759d4c-1703-4dd0-9a75-d6eb260cc964-kube-api-access-qzfjf" (OuterVolumeSpecName: "kube-api-access-qzfjf") pod "b7759d4c-1703-4dd0-9a75-d6eb260cc964" (UID: "b7759d4c-1703-4dd0-9a75-d6eb260cc964"). InnerVolumeSpecName "kube-api-access-qzfjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:53:18.205643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.205600    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7759d4c-1703-4dd0-9a75-d6eb260cc964-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:53:18.205643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.205635    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7759d4c-1703-4dd0-9a75-d6eb260cc964-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:53:18.205643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.205644    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzfjf\" (UniqueName: \"kubernetes.io/projected/b7759d4c-1703-4dd0-9a75-d6eb260cc964-kube-api-access-qzfjf\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:53:18.384493 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.384466    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"]
Sep 09 10:53:18.390676 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.390648    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bbfd5f59/e2e-lifecycle-model-bbfd5f59-kserve-5fc574dfc5-z6252"]
Sep 09 10:53:18.478244 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:18.478204    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7759d4c-1703-4dd0-9a75-d6eb260cc964" path="/var/lib/kubelet/pods/b7759d4c-1703-4dd0-9a75-d6eb260cc964/volumes"
Sep 09 10:53:25.604752 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.604715    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"]
Sep 09 10:53:25.605340 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605322    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc05b8b-81a9-48bc-a8c3-a6daf3427839" containerName="mt-curl-7935-65f57e"
Sep 09 10:53:25.605406 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605342    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc05b8b-81a9-48bc-a8c3-a6daf3427839" containerName="mt-curl-7935-65f57e"
Sep 09 10:53:25.605406 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605380    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7759d4c-1703-4dd0-9a75-d6eb260cc964" containerName="main"
Sep 09 10:53:25.605406 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605388    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7759d4c-1703-4dd0-9a75-d6eb260cc964" containerName="main"
Sep 09 10:53:25.605573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605407    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762278ff-3d65-494f-bafa-56c0ca1f91a4" containerName="main"
Sep 09 10:53:25.605573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605421    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="762278ff-3d65-494f-bafa-56c0ca1f91a4" containerName="main"
Sep 09 10:53:25.605573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605526    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7759d4c-1703-4dd0-9a75-d6eb260cc964" containerName="main"
Sep 09 10:53:25.605573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605541    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="762278ff-3d65-494f-bafa-56c0ca1f91a4" containerName="main"
Sep 09 10:53:25.605573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.605552    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc05b8b-81a9-48bc-a8c3-a6daf3427839" containerName="mt-curl-7935-65f57e"
Sep 09 10:53:25.610295 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.610269    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.613082 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.613059    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"openshift-service-ca.crt\""
Sep 09 10:53:25.613771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.613081    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"default-dockercfg-4nnfx\""
Sep 09 10:53:25.613771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.613121    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"sub-test-model-535b75-kserve-self-signed-certs\""
Sep 09 10:53:25.613771 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.613159    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-535b75\"/\"kube-root-ca.crt\""
Sep 09 10:53:25.616437 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.616413    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"]
Sep 09 10:53:25.778476 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.778442    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45524c7a-ec04-4956-8724-d375b67b421a-tls-certs\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.778476 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.778490    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-model-cache\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.778694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.778518    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjdv\" (UniqueName: \"kubernetes.io/projected/45524c7a-ec04-4956-8724-d375b67b421a-kube-api-access-9zjdv\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.778694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.778570    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-home\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.778694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.778629    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-tmp-dir\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.778694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.778665    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-dshm\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.879911 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.879861    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-home\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.880071 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.879934    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-tmp-dir\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.880131 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.880119    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-dshm\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.880169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.880154    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45524c7a-ec04-4956-8724-d375b67b421a-tls-certs\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.880206 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.880198    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-model-cache\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.880249 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.880240    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjdv\" (UniqueName: \"kubernetes.io/projected/45524c7a-ec04-4956-8724-d375b67b421a-kube-api-access-9zjdv\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.880381 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.880353    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-home\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.880466 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.880375    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-tmp-dir\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.880628 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.880598    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-model-cache\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.882470 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.882451    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-dshm\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.882720 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.882703    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45524c7a-ec04-4956-8724-d375b67b421a-tls-certs\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.888491 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.888472    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjdv\" (UniqueName: \"kubernetes.io/projected/45524c7a-ec04-4956-8724-d375b67b421a-kube-api-access-9zjdv\") pod \"sub-test-model-535b75-kserve-6884b99685-8qkzd\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") " pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:25.922567 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:25.922541    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:26.200201 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:26.200173    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"]
Sep 09 10:53:26.202160 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:53:26.202132    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45524c7a_ec04_4956_8724_d375b67b421a.slice/crio-10250b01c1b2ff0481bcea2f84bca3b3c2247c97a3840bf46a05998991305155 WatchSource:0}: Error finding container 10250b01c1b2ff0481bcea2f84bca3b3c2247c97a3840bf46a05998991305155: Status 404 returned error can't find the container with id 10250b01c1b2ff0481bcea2f84bca3b3c2247c97a3840bf46a05998991305155
Sep 09 10:53:27.118270 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:27.118233    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd" event={"ID":"45524c7a-ec04-4956-8724-d375b67b421a","Type":"ContainerStarted","Data":"980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69"}
Sep 09 10:53:27.118642 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:27.118277    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd" event={"ID":"45524c7a-ec04-4956-8724-d375b67b421a","Type":"ContainerStarted","Data":"10250b01c1b2ff0481bcea2f84bca3b3c2247c97a3840bf46a05998991305155"}
Sep 09 10:53:27.137520 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:27.137464    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd" podStartSLOduration=2.137447969 podStartE2EDuration="2.137447969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:53:27.134157442 +0000 UTC m=+1649.189716504" watchObservedRunningTime="2026-09-09 10:53:27.137447969 +0000 UTC m=+1649.193007018"
Sep 09 10:53:30.411081 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:30.411042    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:53:35.923562 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:35.923525    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:35.923973 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:35.923582    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:35.936308 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:35.936284    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:36.186395 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:36.186316    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:53:52.605223 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:53:52.605180    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:54:05.315997 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.315964    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7935-0cca4c"]
Sep 09 10:54:05.319597 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.319581    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:05.328555 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.328536    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dqfgn\""
Sep 09 10:54:05.333543 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.333519    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7935-0cca4c"]
Sep 09 10:54:05.474981 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.474946    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8tvh\" (UniqueName: \"kubernetes.io/projected/22bc602d-7c7a-406a-9f66-6236b33b99bb-kube-api-access-t8tvh\") pod \"mt-curl-7935-0cca4c\" (UID: \"22bc602d-7c7a-406a-9f66-6236b33b99bb\") " pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:05.576436 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.576333    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8tvh\" (UniqueName: \"kubernetes.io/projected/22bc602d-7c7a-406a-9f66-6236b33b99bb-kube-api-access-t8tvh\") pod \"mt-curl-7935-0cca4c\" (UID: \"22bc602d-7c7a-406a-9f66-6236b33b99bb\") " pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:05.585158 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.585127    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8tvh\" (UniqueName: \"kubernetes.io/projected/22bc602d-7c7a-406a-9f66-6236b33b99bb-kube-api-access-t8tvh\") pod \"mt-curl-7935-0cca4c\" (UID: \"22bc602d-7c7a-406a-9f66-6236b33b99bb\") " pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:05.628281 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.628256    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:05.899545 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.899504    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7935-0cca4c"]
Sep 09 10:54:05.904637 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:05.904618    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 10:54:06.357946 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:06.357884    2586 generic.go:356] "Generic (PLEG): container finished" podID="22bc602d-7c7a-406a-9f66-6236b33b99bb" containerID="12ffd77c93087ddb4f05f7648708c7f81791729d7699681b0b3e5860f0bd9f0a" exitCode=0
Sep 09 10:54:06.357946 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:06.357927    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7935-0cca4c" event={"ID":"22bc602d-7c7a-406a-9f66-6236b33b99bb","Type":"ContainerDied","Data":"12ffd77c93087ddb4f05f7648708c7f81791729d7699681b0b3e5860f0bd9f0a"}
Sep 09 10:54:06.358338 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:06.357965    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7935-0cca4c" event={"ID":"22bc602d-7c7a-406a-9f66-6236b33b99bb","Type":"ContainerStarted","Data":"ab6158fa704f410c2a4e5857206a3aa1307d0a49245b67fc15a5a206bcfede4f"}
Sep 09 10:54:07.498965 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.498939    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:07.534757 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.534731    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7935-0cca4c_22bc602d-7c7a-406a-9f66-6236b33b99bb/mt-curl-7935-0cca4c/0.log"
Sep 09 10:54:07.558336 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.558309    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7935-0cca4c"]
Sep 09 10:54:07.562606 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.562578    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7935-0cca4c"]
Sep 09 10:54:07.595598 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.595569    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8tvh\" (UniqueName: \"kubernetes.io/projected/22bc602d-7c7a-406a-9f66-6236b33b99bb-kube-api-access-t8tvh\") pod \"22bc602d-7c7a-406a-9f66-6236b33b99bb\" (UID: \"22bc602d-7c7a-406a-9f66-6236b33b99bb\") "
Sep 09 10:54:07.597612 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.597583    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22bc602d-7c7a-406a-9f66-6236b33b99bb-kube-api-access-t8tvh" (OuterVolumeSpecName: "kube-api-access-t8tvh") pod "22bc602d-7c7a-406a-9f66-6236b33b99bb" (UID: "22bc602d-7c7a-406a-9f66-6236b33b99bb"). InnerVolumeSpecName "kube-api-access-t8tvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:54:07.683137 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.683063    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7935-7db1cd"]
Sep 09 10:54:07.683518 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.683505    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22bc602d-7c7a-406a-9f66-6236b33b99bb" containerName="mt-curl-7935-0cca4c"
Sep 09 10:54:07.683562 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.683520    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="22bc602d-7c7a-406a-9f66-6236b33b99bb" containerName="mt-curl-7935-0cca4c"
Sep 09 10:54:07.683614 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.683604    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="22bc602d-7c7a-406a-9f66-6236b33b99bb" containerName="mt-curl-7935-0cca4c"
Sep 09 10:54:07.686726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.686710    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-7db1cd"
Sep 09 10:54:07.692643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.692614    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7935-7db1cd"]
Sep 09 10:54:07.696537 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.696514    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8tvh\" (UniqueName: \"kubernetes.io/projected/22bc602d-7c7a-406a-9f66-6236b33b99bb-kube-api-access-t8tvh\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:54:07.797312 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.797274    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz86j\" (UniqueName: \"kubernetes.io/projected/6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238-kube-api-access-tz86j\") pod \"mt-curl-7935-7db1cd\" (UID: \"6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238\") " pod="opendatahub/mt-curl-7935-7db1cd"
Sep 09 10:54:07.898104 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.898070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz86j\" (UniqueName: \"kubernetes.io/projected/6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238-kube-api-access-tz86j\") pod \"mt-curl-7935-7db1cd\" (UID: \"6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238\") " pod="opendatahub/mt-curl-7935-7db1cd"
Sep 09 10:54:07.908745 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.908712    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz86j\" (UniqueName: \"kubernetes.io/projected/6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238-kube-api-access-tz86j\") pod \"mt-curl-7935-7db1cd\" (UID: \"6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238\") " pod="opendatahub/mt-curl-7935-7db1cd"
Sep 09 10:54:07.997437 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:07.997367    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-7db1cd"
Sep 09 10:54:08.274514 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:08.274476    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7935-7db1cd"]
Sep 09 10:54:08.276082 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:54:08.276060    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d51e7cc_4211_4ef4_b4e9_7ccbf1e42238.slice/crio-a08bfcc346123ce23fd4a5d82280e44f9c108fa3de19c810461998fdcdc0fc41 WatchSource:0}: Error finding container a08bfcc346123ce23fd4a5d82280e44f9c108fa3de19c810461998fdcdc0fc41: Status 404 returned error can't find the container with id a08bfcc346123ce23fd4a5d82280e44f9c108fa3de19c810461998fdcdc0fc41
Sep 09 10:54:08.372172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:08.372147    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab6158fa704f410c2a4e5857206a3aa1307d0a49245b67fc15a5a206bcfede4f"
Sep 09 10:54:08.372172 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:08.372146    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:08.373517 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:08.373489    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7935-7db1cd" event={"ID":"6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238","Type":"ContainerStarted","Data":"a08bfcc346123ce23fd4a5d82280e44f9c108fa3de19c810461998fdcdc0fc41"}
Sep 09 10:54:08.382835 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:54:08.382809    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7935-0cca4c\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="22bc602d-7c7a-406a-9f66-6236b33b99bb" pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:08.474091 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:54:08.474056    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7935-0cca4c\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="22bc602d-7c7a-406a-9f66-6236b33b99bb" pod="opendatahub/mt-curl-7935-0cca4c"
Sep 09 10:54:08.476448 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:08.476424    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22bc602d-7c7a-406a-9f66-6236b33b99bb" path="/var/lib/kubelet/pods/22bc602d-7c7a-406a-9f66-6236b33b99bb/volumes"
Sep 09 10:54:09.380519 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:09.380487    2586 generic.go:356] "Generic (PLEG): container finished" podID="6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238" containerID="e2e3482491ca2c2d968d2a8e16c6983f686a4a740189e2254f555390d3b64d61" exitCode=0
Sep 09 10:54:09.381025 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:09.380582    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7935-7db1cd" event={"ID":"6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238","Type":"ContainerDied","Data":"e2e3482491ca2c2d968d2a8e16c6983f686a4a740189e2254f555390d3b64d61"}
Sep 09 10:54:10.523153 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:10.523107    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-7db1cd"
Sep 09 10:54:10.544221 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:10.544191    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7935-7db1cd_6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238/mt-curl-7935-7db1cd/0.log"
Sep 09 10:54:10.566357 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:10.566327    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7935-7db1cd"]
Sep 09 10:54:10.570220 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:10.570194    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7935-7db1cd"]
Sep 09 10:54:10.625444 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:10.625413    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz86j\" (UniqueName: \"kubernetes.io/projected/6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238-kube-api-access-tz86j\") pod \"6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238\" (UID: \"6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238\") "
Sep 09 10:54:10.627517 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:10.627493    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238-kube-api-access-tz86j" (OuterVolumeSpecName: "kube-api-access-tz86j") pod "6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238" (UID: "6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238"). InnerVolumeSpecName "kube-api-access-tz86j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:54:10.727230 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:10.727142    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz86j\" (UniqueName: \"kubernetes.io/projected/6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238-kube-api-access-tz86j\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:54:11.393931 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:11.393875    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7935-7db1cd"
Sep 09 10:54:11.393931 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:11.393907    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a08bfcc346123ce23fd4a5d82280e44f9c108fa3de19c810461998fdcdc0fc41"
Sep 09 10:54:11.404026 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:54:11.403996    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7935-7db1cd\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238" pod="opendatahub/mt-curl-7935-7db1cd"
Sep 09 10:54:11.404150 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:11.404132    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:54:12.475043 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:12.475006    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238" path="/var/lib/kubelet/pods/6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238/volumes"
Sep 09 10:54:13.601106 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:13.601074    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"]
Sep 09 10:54:24.308613 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:24.308532    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:54:28.620613 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.620570    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd" podUID="45524c7a-ec04-4956-8724-d375b67b421a" containerName="main" containerID="cri-o://980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69" gracePeriod=45
Sep 09 10:54:28.865490 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.865465    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:54:28.990536 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990418    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-dshm\") pod \"45524c7a-ec04-4956-8724-d375b67b421a\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") "
Sep 09 10:54:28.990536 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990500    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zjdv\" (UniqueName: \"kubernetes.io/projected/45524c7a-ec04-4956-8724-d375b67b421a-kube-api-access-9zjdv\") pod \"45524c7a-ec04-4956-8724-d375b67b421a\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") "
Sep 09 10:54:28.990536 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990526    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-tmp-dir\") pod \"45524c7a-ec04-4956-8724-d375b67b421a\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") "
Sep 09 10:54:28.990833 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990555    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-home\") pod \"45524c7a-ec04-4956-8724-d375b67b421a\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") "
Sep 09 10:54:28.990833 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990581    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45524c7a-ec04-4956-8724-d375b67b421a-tls-certs\") pod \"45524c7a-ec04-4956-8724-d375b67b421a\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") "
Sep 09 10:54:28.990833 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990663    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-model-cache\") pod \"45524c7a-ec04-4956-8724-d375b67b421a\" (UID: \"45524c7a-ec04-4956-8724-d375b67b421a\") "
Sep 09 10:54:28.991085 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990834    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-home" (OuterVolumeSpecName: "home") pod "45524c7a-ec04-4956-8724-d375b67b421a" (UID: "45524c7a-ec04-4956-8724-d375b67b421a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:54:28.991085 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990842    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45524c7a-ec04-4956-8724-d375b67b421a" (UID: "45524c7a-ec04-4956-8724-d375b67b421a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:54:28.991085 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.990978    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-model-cache" (OuterVolumeSpecName: "model-cache") pod "45524c7a-ec04-4956-8724-d375b67b421a" (UID: "45524c7a-ec04-4956-8724-d375b67b421a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:54:28.991251 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.991076    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:54:28.991251 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.991104    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:54:28.991251 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.991112    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:54:28.992808 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.992783    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45524c7a-ec04-4956-8724-d375b67b421a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45524c7a-ec04-4956-8724-d375b67b421a" (UID: "45524c7a-ec04-4956-8724-d375b67b421a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:54:28.993185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.993156    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-dshm" (OuterVolumeSpecName: "dshm") pod "45524c7a-ec04-4956-8724-d375b67b421a" (UID: "45524c7a-ec04-4956-8724-d375b67b421a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:54:28.993185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:28.993158    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45524c7a-ec04-4956-8724-d375b67b421a-kube-api-access-9zjdv" (OuterVolumeSpecName: "kube-api-access-9zjdv") pod "45524c7a-ec04-4956-8724-d375b67b421a" (UID: "45524c7a-ec04-4956-8724-d375b67b421a"). InnerVolumeSpecName "kube-api-access-9zjdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:54:29.092609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.092569    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45524c7a-ec04-4956-8724-d375b67b421a-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:54:29.092609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.092602    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjdv\" (UniqueName: \"kubernetes.io/projected/45524c7a-ec04-4956-8724-d375b67b421a-kube-api-access-9zjdv\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:54:29.092609 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.092613    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45524c7a-ec04-4956-8724-d375b67b421a-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:54:29.505522 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.505487    2586 generic.go:356] "Generic (PLEG): container finished" podID="45524c7a-ec04-4956-8724-d375b67b421a" containerID="980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69" exitCode=0
Sep 09 10:54:29.505694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.505538    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"
Sep 09 10:54:29.505694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.505571    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd" event={"ID":"45524c7a-ec04-4956-8724-d375b67b421a","Type":"ContainerDied","Data":"980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69"}
Sep 09 10:54:29.505694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.505618    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd" event={"ID":"45524c7a-ec04-4956-8724-d375b67b421a","Type":"ContainerDied","Data":"10250b01c1b2ff0481bcea2f84bca3b3c2247c97a3840bf46a05998991305155"}
Sep 09 10:54:29.505694 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.505642    2586 scope.go:117] "RemoveContainer" containerID="980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69"
Sep 09 10:54:29.521413 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.521381    2586 scope.go:117] "RemoveContainer" containerID="980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69"
Sep 09 10:54:29.522300 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:54:29.522029    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69\": container with ID starting with 980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69 not found: ID does not exist" containerID="980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69"
Sep 09 10:54:29.522300 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.522070    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69"} err="failed to get container status \"980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69\": rpc error: code = NotFound desc = could not find container \"980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69\": container with ID starting with 980218f72478ccd17beca8872554079ee8f84902f259afbeac243ae865de7a69 not found: ID does not exist"
Sep 09 10:54:29.535955 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.535928    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"]
Sep 09 10:54:29.540007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:29.539984    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-535b75/sub-test-model-535b75-kserve-6884b99685-8qkzd"]
Sep 09 10:54:30.475695 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:30.475658    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45524c7a-ec04-4956-8724-d375b67b421a" path="/var/lib/kubelet/pods/45524c7a-ec04-4956-8724-d375b67b421a/volumes"
Sep 09 10:54:33.212390 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:33.212354    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:54:53.409526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:54:53.409486    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:55:11.262656 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.262626    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"]
Sep 09 10:55:11.263087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.263073    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45524c7a-ec04-4956-8724-d375b67b421a" containerName="main"
Sep 09 10:55:11.263087 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.263089    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="45524c7a-ec04-4956-8724-d375b67b421a" containerName="main"
Sep 09 10:55:11.263179 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.263105    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238" containerName="mt-curl-7935-7db1cd"
Sep 09 10:55:11.263179 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.263111    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238" containerName="mt-curl-7935-7db1cd"
Sep 09 10:55:11.263179 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.263176    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="45524c7a-ec04-4956-8724-d375b67b421a" containerName="main"
Sep 09 10:55:11.263275 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.263185    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d51e7cc-4211-4ef4-b4e9-7ccbf1e42238" containerName="mt-curl-7935-7db1cd"
Sep 09 10:55:11.266298 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.266280    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.268753 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.268723    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-798524-openshift-default-dockercfg-7fjfx\""
Sep 09 10:55:11.282216 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.282185    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"]
Sep 09 10:55:11.400990 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.400950    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023fa7f4-453e-4677-9fdb-c32618a7fadc-istiod-ca-cert\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.401169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.401005    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-socket\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.401169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.401038    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-certs\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.401169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.401076    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-data\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.401169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.401099    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxjc2\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-kube-api-access-rxjc2\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.401169 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.401162    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-podinfo\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.401345 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.401186    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-envoy\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.401345 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.401204    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-credential-socket\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.401345 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.401224    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-token\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.501784 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.501746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-token\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.501802    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023fa7f4-453e-4677-9fdb-c32618a7fadc-istiod-ca-cert\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.501834    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-socket\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.501862    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-certs\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502193 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502008    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-data\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502193 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502080    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxjc2\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-kube-api-access-rxjc2\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502193 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502166    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-podinfo\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502196    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-envoy\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502230    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-credential-socket\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502275    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-socket\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502314    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-certs\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502582 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502450    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-data\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502642 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502589    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-credential-socket\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.502642 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.502614    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023fa7f4-453e-4677-9fdb-c32618a7fadc-istiod-ca-cert\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.504451 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.504432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-envoy\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.504673 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.504653    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-podinfo\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.512520 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.512486    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-token\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.512719 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.512669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxjc2\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-kube-api-access-rxjc2\") pod \"e2e-api-b-798524-openshift-default-79ff884d88-wjjmx\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") " pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.578738 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.578700    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:11.861467 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.861441    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"]
Sep 09 10:55:11.862949 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:55:11.862919    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023fa7f4_453e_4677_9fdb_c32618a7fadc.slice/crio-bc505e265b28520e233c5c6d7942874014e9a3cecfbea199409de63c01e8b8a7 WatchSource:0}: Error finding container bc505e265b28520e233c5c6d7942874014e9a3cecfbea199409de63c01e8b8a7: Status 404 returned error can't find the container with id bc505e265b28520e233c5c6d7942874014e9a3cecfbea199409de63c01e8b8a7
Sep 09 10:55:11.865122 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.865087    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:11.865211 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.865174    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:11.865211 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:11.865202    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:12.782940 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:12.782879    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" event={"ID":"023fa7f4-453e-4677-9fdb-c32618a7fadc","Type":"ContainerStarted","Data":"3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8"}
Sep 09 10:55:12.783350 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:12.782969    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" event={"ID":"023fa7f4-453e-4677-9fdb-c32618a7fadc","Type":"ContainerStarted","Data":"bc505e265b28520e233c5c6d7942874014e9a3cecfbea199409de63c01e8b8a7"}
Sep 09 10:55:12.806568 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:12.806513    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podStartSLOduration=1.80649755 podStartE2EDuration="1.80649755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:55:12.804488145 +0000 UTC m=+1754.860047195" watchObservedRunningTime="2026-09-09 10:55:12.80649755 +0000 UTC m=+1754.862056599"
Sep 09 10:55:13.579626 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:13.579578    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:13.585182 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:13.585158    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:13.789872 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:13.789844    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:13.790936 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:13.790919    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:15.903759 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:15.903719    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"]
Sep 09 10:55:15.907784 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:15.907763    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:15.910075 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:15.910050    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-798524-serving-cert\""
Sep 09 10:55:15.920075 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:15.920047    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"]
Sep 09 10:55:16.048198 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.048155    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.048393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.048210    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8p7s\" (UniqueName: \"kubernetes.io/projected/1870e3ee-9077-4b13-8c0e-150d9d72b65f-kube-api-access-w8p7s\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.048393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.048326    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-metrics-certs\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.048472 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.048408    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1870e3ee-9077-4b13-8c0e-150d9d72b65f-openshift-service-ca\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.149294 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.149253    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1870e3ee-9077-4b13-8c0e-150d9d72b65f-openshift-service-ca\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.149488 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.149304    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.149488 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.149326    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8p7s\" (UniqueName: \"kubernetes.io/projected/1870e3ee-9077-4b13-8c0e-150d9d72b65f-kube-api-access-w8p7s\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.149488 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.149376    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-metrics-certs\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.149488 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:55:16.149442    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-798524-serving-cert: secret "maas-api-e2e-api-b-798524-serving-cert" not found
Sep 09 10:55:16.149697 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:55:16.149521    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls podName:1870e3ee-9077-4b13-8c0e-150d9d72b65f nodeName:}" failed. No retries permitted until 2026-09-09 10:55:16.649498114 +0000 UTC m=+1758.705057149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls") pod "maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f" (UID: "1870e3ee-9077-4b13-8c0e-150d9d72b65f") : secret "maas-api-e2e-api-b-798524-serving-cert" not found
Sep 09 10:55:16.150180 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.150154    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1870e3ee-9077-4b13-8c0e-150d9d72b65f-openshift-service-ca\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.151815 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.151798    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-metrics-certs\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.157693 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.157641    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8p7s\" (UniqueName: \"kubernetes.io/projected/1870e3ee-9077-4b13-8c0e-150d9d72b65f-kube-api-access-w8p7s\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.655401 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.655358    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.657809 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.657780    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls\") pod \"maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:16.822391 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:16.822356    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:17.109942 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:17.109916    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"]
Sep 09 10:55:17.112505 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:55:17.112473    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1870e3ee_9077_4b13_8c0e_150d9d72b65f.slice/crio-24ff7fcf91b47235096d04180fb8c80aa6b62855cea7a34310688bdadc6a492c WatchSource:0}: Error finding container 24ff7fcf91b47235096d04180fb8c80aa6b62855cea7a34310688bdadc6a492c: Status 404 returned error can't find the container with id 24ff7fcf91b47235096d04180fb8c80aa6b62855cea7a34310688bdadc6a492c
Sep 09 10:55:17.817526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:17.817491    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f" event={"ID":"1870e3ee-9077-4b13-8c0e-150d9d72b65f","Type":"ContainerStarted","Data":"af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962"}
Sep 09 10:55:17.817696 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:17.817541    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f" event={"ID":"1870e3ee-9077-4b13-8c0e-150d9d72b65f","Type":"ContainerStarted","Data":"24ff7fcf91b47235096d04180fb8c80aa6b62855cea7a34310688bdadc6a492c"}
Sep 09 10:55:17.817696 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:17.817681    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:17.838112 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:17.838006    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f" podStartSLOduration=2.370751124 podStartE2EDuration="2.837987928s" totalImagesPullingTime="467.236804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:55:17.835318183 +0000 UTC m=+1759.890877237" watchObservedRunningTime="2026-09-09 10:55:17.837987928 +0000 UTC m=+1759.893546979"
Sep 09 10:55:23.829553 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:23.829520    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:55:26.714870 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:26.714835    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:55:36.938715 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:36.938681    2586 generic.go:356] "Generic (PLEG): container finished" podID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerID="3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8" exitCode=0
Sep 09 10:55:36.939141 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:36.938752    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" event={"ID":"023fa7f4-453e-4677-9fdb-c32618a7fadc","Type":"ContainerDied","Data":"3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8"}
Sep 09 10:55:36.939198 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:36.939184    2586 scope.go:117] "RemoveContainer" containerID="3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8"
Sep 09 10:55:36.940308 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:36.940283    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:36.940382 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:36.940362    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:36.940422 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:36.940396    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:55:37.948784 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:37.948744    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" event={"ID":"023fa7f4-453e-4677-9fdb-c32618a7fadc","Type":"ContainerStarted","Data":"152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04"}
Sep 09 10:55:38.579322 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:38.579290    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:38.579518 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:38.579489    2586 patch_prober.go:28] interesting pod/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 10:55:38.579568 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:38.579531    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 10:55:39.580053 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:39.580014    2586 patch_prober.go:28] interesting pod/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 10:55:39.580533 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:39.580101    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 10:55:40.579880 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:40.579843    2586 patch_prober.go:28] interesting pod/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 10:55:40.580097 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:40.579927    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 10:55:41.579807 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:41.579761    2586 patch_prober.go:28] interesting pod/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 10:55:41.580049 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:41.579837    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 10:55:42.579661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:42.579630    2586 patch_prober.go:28] interesting pod/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 10:55:42.580067 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:42.579681    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 10:55:43.579800 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:43.579764    2586 patch_prober.go:28] interesting pod/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 10:55:43.580201 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:43.579823    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 10:55:43.789994 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:43.789956    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:44.583523 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:44.583486    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:44.584607 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:44.584588    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:55:50.665869 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:55:50.665791    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:56:05.408550 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:05.408511    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:56:18.610784 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:18.610747    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:56:21.394212 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.394177    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"]
Sep 09 10:56:21.394635 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.394424    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f" podUID="1870e3ee-9077-4b13-8c0e-150d9d72b65f" containerName="maas-api" containerID="cri-o://af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962" gracePeriod=30
Sep 09 10:56:21.650786 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.650698    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:56:21.710487 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.710440    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8p7s\" (UniqueName: \"kubernetes.io/projected/1870e3ee-9077-4b13-8c0e-150d9d72b65f-kube-api-access-w8p7s\") pod \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") "
Sep 09 10:56:21.710689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.710502    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-metrics-certs\") pod \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") "
Sep 09 10:56:21.710689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.710571    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls\") pod \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") "
Sep 09 10:56:21.710689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.710679    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1870e3ee-9077-4b13-8c0e-150d9d72b65f-openshift-service-ca\") pod \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\" (UID: \"1870e3ee-9077-4b13-8c0e-150d9d72b65f\") "
Sep 09 10:56:21.711074 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.711042    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1870e3ee-9077-4b13-8c0e-150d9d72b65f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1870e3ee-9077-4b13-8c0e-150d9d72b65f" (UID: "1870e3ee-9077-4b13-8c0e-150d9d72b65f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:56:21.712790 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.712765    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1870e3ee-9077-4b13-8c0e-150d9d72b65f" (UID: "1870e3ee-9077-4b13-8c0e-150d9d72b65f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:56:21.713083 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.713064    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1870e3ee-9077-4b13-8c0e-150d9d72b65f" (UID: "1870e3ee-9077-4b13-8c0e-150d9d72b65f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:56:21.713167 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.713089    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1870e3ee-9077-4b13-8c0e-150d9d72b65f-kube-api-access-w8p7s" (OuterVolumeSpecName: "kube-api-access-w8p7s") pod "1870e3ee-9077-4b13-8c0e-150d9d72b65f" (UID: "1870e3ee-9077-4b13-8c0e-150d9d72b65f"). InnerVolumeSpecName "kube-api-access-w8p7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:21.812399 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.812365    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:21.812399 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.812394    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1870e3ee-9077-4b13-8c0e-150d9d72b65f-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:21.812399 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.812405    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8p7s\" (UniqueName: \"kubernetes.io/projected/1870e3ee-9077-4b13-8c0e-150d9d72b65f-kube-api-access-w8p7s\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:21.812641 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:21.812413    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870e3ee-9077-4b13-8c0e-150d9d72b65f-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:22.237238 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.237206    2586 generic.go:356] "Generic (PLEG): container finished" podID="1870e3ee-9077-4b13-8c0e-150d9d72b65f" containerID="af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962" exitCode=0
Sep 09 10:56:22.237460 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.237247    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"
Sep 09 10:56:22.237460 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.237300    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f" event={"ID":"1870e3ee-9077-4b13-8c0e-150d9d72b65f","Type":"ContainerDied","Data":"af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962"}
Sep 09 10:56:22.237460 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.237361    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f" event={"ID":"1870e3ee-9077-4b13-8c0e-150d9d72b65f","Type":"ContainerDied","Data":"24ff7fcf91b47235096d04180fb8c80aa6b62855cea7a34310688bdadc6a492c"}
Sep 09 10:56:22.237460 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.237394    2586 scope.go:117] "RemoveContainer" containerID="af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962"
Sep 09 10:56:22.248404 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.248388    2586 scope.go:117] "RemoveContainer" containerID="af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962"
Sep 09 10:56:22.248779 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:56:22.248755    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962\": container with ID starting with af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962 not found: ID does not exist" containerID="af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962"
Sep 09 10:56:22.248828 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.248789    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962"} err="failed to get container status \"af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962\": rpc error: code = NotFound desc = could not find container \"af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962\": container with ID starting with af49965598384f7b3121f9a1dea1787c10a895b437308f0d23db88ff22d3b962 not found: ID does not exist"
Sep 09 10:56:22.263723 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.263693    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"]
Sep 09 10:56:22.268825 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.268804    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-798524-5ddf4dc6bf-tnz5f"]
Sep 09 10:56:22.475185 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:22.475150    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1870e3ee-9077-4b13-8c0e-150d9d72b65f" path="/var/lib/kubelet/pods/1870e3ee-9077-4b13-8c0e-150d9d72b65f/volumes"
Sep 09 10:56:28.850196 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:28.850159    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"]
Sep 09 10:56:28.850652 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:28.850423    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy" containerID="cri-o://152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04" gracePeriod=30
Sep 09 10:56:29.411076 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:29.411043    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:56:29.586631 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:29.586588    2586 patch_prober.go:28] interesting pod/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 10:56:29.586776 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:29.586651    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 10:56:34.103420 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.103393    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:56:34.233825 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.233734    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-podinfo\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.233825 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.233791    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023fa7f4-453e-4677-9fdb-c32618a7fadc-istiod-ca-cert\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.234098 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.233831    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-certs\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.234098 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.233858    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-token\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.234098 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234005    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-data\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.234098 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234079    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-socket\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.234389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234140    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxjc2\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-kube-api-access-rxjc2\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.234389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234168    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:34.234389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234217    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-envoy\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.234389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234232    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-data" (OuterVolumeSpecName: "istio-data") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:34.234389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234257    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023fa7f4-453e-4677-9fdb-c32618a7fadc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:56:34.234389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234268    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-credential-socket\") pod \"023fa7f4-453e-4677-9fdb-c32618a7fadc\" (UID: \"023fa7f4-453e-4677-9fdb-c32618a7fadc\") "
Sep 09 10:56:34.235393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234393    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:34.235393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234629    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:34.235393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234697    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.235393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234720    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023fa7f4-453e-4677-9fdb-c32618a7fadc-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.235393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234733    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.235393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.234745    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.236058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.236034    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:56:34.236203 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.236184    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-token" (OuterVolumeSpecName: "istio-token") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:34.236316 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.236296    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:34.236526 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.236511    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-kube-api-access-rxjc2" (OuterVolumeSpecName: "kube-api-access-rxjc2") pod "023fa7f4-453e-4677-9fdb-c32618a7fadc" (UID: "023fa7f4-453e-4677-9fdb-c32618a7fadc"). InnerVolumeSpecName "kube-api-access-rxjc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:34.327390 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.327356    2586 generic.go:356] "Generic (PLEG): container finished" podID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerID="152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04" exitCode=0
Sep 09 10:56:34.327572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.327399    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"
Sep 09 10:56:34.327572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.327437    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" event={"ID":"023fa7f4-453e-4677-9fdb-c32618a7fadc","Type":"ContainerDied","Data":"152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04"}
Sep 09 10:56:34.327572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.327495    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx" event={"ID":"023fa7f4-453e-4677-9fdb-c32618a7fadc","Type":"ContainerDied","Data":"bc505e265b28520e233c5c6d7942874014e9a3cecfbea199409de63c01e8b8a7"}
Sep 09 10:56:34.327572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.327518    2586 scope.go:117] "RemoveContainer" containerID="152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04"
Sep 09 10:56:34.335560 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.335539    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.335560 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.335561    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.335763 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.335570    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxjc2\" (UniqueName: \"kubernetes.io/projected/023fa7f4-453e-4677-9fdb-c32618a7fadc-kube-api-access-rxjc2\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.335763 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.335578    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.335763 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.335588    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023fa7f4-453e-4677-9fdb-c32618a7fadc-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:34.338435 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.338419    2586 scope.go:117] "RemoveContainer" containerID="3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8"
Sep 09 10:56:34.346854 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.346837    2586 scope.go:117] "RemoveContainer" containerID="152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04"
Sep 09 10:56:34.347140 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:56:34.347121    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04\": container with ID starting with 152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04 not found: ID does not exist" containerID="152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04"
Sep 09 10:56:34.347184 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.347147    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04"} err="failed to get container status \"152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04\": rpc error: code = NotFound desc = could not find container \"152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04\": container with ID starting with 152af228c1d50d17e04f20f9e0d379fa5430bec5f4c0a58655603e45cbf0ef04 not found: ID does not exist"
Sep 09 10:56:34.347184 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.347164    2586 scope.go:117] "RemoveContainer" containerID="3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8"
Sep 09 10:56:34.347409 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:56:34.347391    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8\": container with ID starting with 3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8 not found: ID does not exist" containerID="3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8"
Sep 09 10:56:34.347450 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.347415    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8"} err="failed to get container status \"3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8\": rpc error: code = NotFound desc = could not find container \"3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8\": container with ID starting with 3f3ee2ad9dba8c8c485543714a2a7dcd766d015bc4e72bf87a464fd09ad14cd8 not found: ID does not exist"
Sep 09 10:56:34.351871 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.351849    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"]
Sep 09 10:56:34.356875 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.356854    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-798524-openshift-default-79ff884d88-wjjmx"]
Sep 09 10:56:34.477495 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:34.477453    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" path="/var/lib/kubelet/pods/023fa7f4-453e-4677-9fdb-c32618a7fadc/volumes"
Sep 09 10:56:42.444970 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:42.444594    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:56:47.295154 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:47.295104    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"]
Sep 09 10:56:47.295574 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:47.295386    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy" containerID="cri-o://33079d934cf2146d8218f6d0d5e986283807f3b1425069c577fffac8e55269f7" gracePeriod=30
Sep 09 10:56:52.445642 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.445604    2586 generic.go:356] "Generic (PLEG): container finished" podID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerID="33079d934cf2146d8218f6d0d5e986283807f3b1425069c577fffac8e55269f7" exitCode=0
Sep 09 10:56:52.446076 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.445644    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" event={"ID":"5a6152dd-f966-4d23-9d91-42ab3743b64a","Type":"ContainerDied","Data":"33079d934cf2146d8218f6d0d5e986283807f3b1425069c577fffac8e55269f7"}
Sep 09 10:56:52.446076 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.445709    2586 scope.go:117] "RemoveContainer" containerID="7f324b6e3093a4387fda3268821315e396c3940a173a7d1411056966089016e7"
Sep 09 10:56:52.562164 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.562097    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:56:52.617024 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.616987    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-envoy\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617024 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617028    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-credential-socket\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617064    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-podinfo\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617121    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-token\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617152    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-socket\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617191    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-data\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617228    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-certs\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617251    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6152dd-f966-4d23-9d91-42ab3743b64a-istiod-ca-cert\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617310 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617287    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbhtp\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-kube-api-access-jbhtp\") pod \"5a6152dd-f966-4d23-9d91-42ab3743b64a\" (UID: \"5a6152dd-f966-4d23-9d91-42ab3743b64a\") "
Sep 09 10:56:52.617697 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617349    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.617697 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617589    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.617697 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617628    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6152dd-f966-4d23-9d91-42ab3743b64a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:56:52.617697 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617669    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.617697 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617687    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.617997 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617768    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.617997 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.617784    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.619643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.619611    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:56:52.619993 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.619965    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:56:52.620100 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.620016    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-kube-api-access-jbhtp" (OuterVolumeSpecName: "kube-api-access-jbhtp") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "kube-api-access-jbhtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:52.620100 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.620016    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a6152dd-f966-4d23-9d91-42ab3743b64a" (UID: "5a6152dd-f966-4d23-9d91-42ab3743b64a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:52.718354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.718315    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.718354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.718347    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.718354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.718357    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.718354 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.718367    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6152dd-f966-4d23-9d91-42ab3743b64a-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.718618 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.718375    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhtp\" (UniqueName: \"kubernetes.io/projected/5a6152dd-f966-4d23-9d91-42ab3743b64a-kube-api-access-jbhtp\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.718618 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.718384    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:52.718618 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:52.718392    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6152dd-f966-4d23-9d91-42ab3743b64a-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:53.456686 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:53.456656    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"
Sep 09 10:56:53.457143 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:53.456700    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq" event={"ID":"5a6152dd-f966-4d23-9d91-42ab3743b64a","Type":"ContainerDied","Data":"74e5c02f24de90d091b40f0cfa17300e05841043c08fcb895fa2e65552815902"}
Sep 09 10:56:53.457143 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:53.456761    2586 scope.go:117] "RemoveContainer" containerID="33079d934cf2146d8218f6d0d5e986283807f3b1425069c577fffac8e55269f7"
Sep 09 10:56:53.478715 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:53.478684    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"]
Sep 09 10:56:53.484752 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:53.484723    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-62f893-openshift-default-658578b5c8-rg5mq"]
Sep 09 10:56:54.474691 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:54.474656    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" path="/var/lib/kubelet/pods/5a6152dd-f966-4d23-9d91-42ab3743b64a/volumes"
Sep 09 10:56:58.095451 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.095412    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:56:58.280969 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.280937    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"]
Sep 09 10:56:58.281220 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.281194    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9" podUID="7cef9733-957d-4ee3-84ad-16eac3d4c56c" containerName="maas-api" containerID="cri-o://a4e605fd0e99985c30c4625a8ea9e7990af3409ffee24226bbcda1abb82082b4" gracePeriod=30
Sep 09 10:56:58.493932 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.493882    2586 generic.go:356] "Generic (PLEG): container finished" podID="7cef9733-957d-4ee3-84ad-16eac3d4c56c" containerID="a4e605fd0e99985c30c4625a8ea9e7990af3409ffee24226bbcda1abb82082b4" exitCode=0
Sep 09 10:56:58.494093 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.493957    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9" event={"ID":"7cef9733-957d-4ee3-84ad-16eac3d4c56c","Type":"ContainerDied","Data":"a4e605fd0e99985c30c4625a8ea9e7990af3409ffee24226bbcda1abb82082b4"}
Sep 09 10:56:58.540350 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.540315    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:56:58.575721 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.575692    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls\") pod \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") "
Sep 09 10:56:58.575902 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.575778    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-metrics-certs\") pod \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") "
Sep 09 10:56:58.575985 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.575922    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef9733-957d-4ee3-84ad-16eac3d4c56c-openshift-service-ca\") pod \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") "
Sep 09 10:56:58.575985 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.575955    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rnnl\" (UniqueName: \"kubernetes.io/projected/7cef9733-957d-4ee3-84ad-16eac3d4c56c-kube-api-access-7rnnl\") pod \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\" (UID: \"7cef9733-957d-4ee3-84ad-16eac3d4c56c\") "
Sep 09 10:56:58.576274 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.576246    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cef9733-957d-4ee3-84ad-16eac3d4c56c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7cef9733-957d-4ee3-84ad-16eac3d4c56c" (UID: "7cef9733-957d-4ee3-84ad-16eac3d4c56c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:56:58.576362 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.576335    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef9733-957d-4ee3-84ad-16eac3d4c56c-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:58.577969 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.577940    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7cef9733-957d-4ee3-84ad-16eac3d4c56c" (UID: "7cef9733-957d-4ee3-84ad-16eac3d4c56c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:56:58.578069 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.578000    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7cef9733-957d-4ee3-84ad-16eac3d4c56c" (UID: "7cef9733-957d-4ee3-84ad-16eac3d4c56c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:56:58.578114 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.578077    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cef9733-957d-4ee3-84ad-16eac3d4c56c-kube-api-access-7rnnl" (OuterVolumeSpecName: "kube-api-access-7rnnl") pod "7cef9733-957d-4ee3-84ad-16eac3d4c56c" (UID: "7cef9733-957d-4ee3-84ad-16eac3d4c56c"). InnerVolumeSpecName "kube-api-access-7rnnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:56:58.678007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.677910    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:58.678007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.677953    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rnnl\" (UniqueName: \"kubernetes.io/projected/7cef9733-957d-4ee3-84ad-16eac3d4c56c-kube-api-access-7rnnl\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:58.678007 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.677969    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef9733-957d-4ee3-84ad-16eac3d4c56c-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:56:58.739482 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:58.739452    2586 scope.go:117] "RemoveContainer" containerID="a4e605fd0e99985c30c4625a8ea9e7990af3409ffee24226bbcda1abb82082b4"
Sep 09 10:56:59.499426 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:59.499388    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9" event={"ID":"7cef9733-957d-4ee3-84ad-16eac3d4c56c","Type":"ContainerDied","Data":"1cc5bf7de1a5b28bdf8ee983ec95d85d39f85cc4596d00274cd05047e71d3621"}
Sep 09 10:56:59.499907 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:59.499452    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"
Sep 09 10:56:59.530904 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:59.530858    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"]
Sep 09 10:56:59.535450 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:56:59.535421    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a03d9e-6964cd6ffc-mbqv9"]
Sep 09 10:57:00.476304 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:00.476272    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cef9733-957d-4ee3-84ad-16eac3d4c56c" path="/var/lib/kubelet/pods/7cef9733-957d-4ee3-84ad-16eac3d4c56c/volumes"
Sep 09 10:57:05.849468 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:05.849428    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"]
Sep 09 10:57:05.849937 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:05.849643    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy" containerID="cri-o://f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f" gracePeriod=30
Sep 09 10:57:07.141117 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141080    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"]
Sep 09 10:57:07.141527 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141518    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy"
Sep 09 10:57:07.141573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141530    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy"
Sep 09 10:57:07.141573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141543    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy"
Sep 09 10:57:07.141573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141548    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy"
Sep 09 10:57:07.141573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141555    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1870e3ee-9077-4b13-8c0e-150d9d72b65f" containerName="maas-api"
Sep 09 10:57:07.141573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141561    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1870e3ee-9077-4b13-8c0e-150d9d72b65f" containerName="maas-api"
Sep 09 10:57:07.141573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141571    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141576    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141584    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cef9733-957d-4ee3-84ad-16eac3d4c56c" containerName="maas-api"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141592    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cef9733-957d-4ee3-84ad-16eac3d4c56c" containerName="maas-api"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141606    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141612    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141673    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cef9733-957d-4ee3-84ad-16eac3d4c56c" containerName="maas-api"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141679    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141686    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="023fa7f4-453e-4677-9fdb-c32618a7fadc" containerName="istio-proxy"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141693    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy"
Sep 09 10:57:07.141887 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.141697    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1870e3ee-9077-4b13-8c0e-150d9d72b65f" containerName="maas-api"
Sep 09 10:57:07.144984 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.144964    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.147393 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.147371    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c1bfbb-openshift-default-dockercfg-lt99z\""
Sep 09 10:57:07.160932 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.160904    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"]
Sep 09 10:57:07.260321 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260289    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-credential-socket\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.260519 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260330    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-data\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.260519 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260368    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-certs\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.260519 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-token\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.260667 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260593    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-socket\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.260667 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260632    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlkm\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-kube-api-access-6jlkm\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.260667 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260657    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istiod-ca-cert\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.260776 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260677    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-podinfo\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.260776 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.260737    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-envoy\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.361980 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.361926    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istiod-ca-cert\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.361989    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-podinfo\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362046    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-envoy\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362097    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-credential-socket\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362130    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-data\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362195 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-certs\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362226    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-token\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362319    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-socket\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362489 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362354    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlkm\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-kube-api-access-6jlkm\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362696 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362668    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-certs\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362772 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362740    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-socket\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.362981 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362952    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-data\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.363100 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.362981    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istiod-ca-cert\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.363100 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.363081    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-credential-socket\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.364685 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.364658    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-envoy\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.365020 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.365002    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-podinfo\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.370967 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.370944    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-token\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.371049 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.371029    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlkm\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-kube-api-access-6jlkm\") pod \"e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") " pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.456492 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.456395    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:07.756175 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.756148    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"]
Sep 09 10:57:07.758047 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:57:07.758013    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bc31fa_0ca4_40a8_8c42_fe02414c2c3f.slice/crio-272999905a8127a6f4f54295189300cdccb89ad53be9cc8f382d0a6157b0276b WatchSource:0}: Error finding container 272999905a8127a6f4f54295189300cdccb89ad53be9cc8f382d0a6157b0276b: Status 404 returned error can't find the container with id 272999905a8127a6f4f54295189300cdccb89ad53be9cc8f382d0a6157b0276b
Sep 09 10:57:07.760432 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.760391    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:07.760514 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.760486    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:07.760563 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:07.760533    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:08.561572 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:08.561537    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" event={"ID":"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f","Type":"ContainerStarted","Data":"4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8"}
Sep 09 10:57:08.562031 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:08.561595    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" event={"ID":"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f","Type":"ContainerStarted","Data":"272999905a8127a6f4f54295189300cdccb89ad53be9cc8f382d0a6157b0276b"}
Sep 09 10:57:08.587156 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:08.587099    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" podStartSLOduration=1.587082758 podStartE2EDuration="1.587082758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:57:08.585057924 +0000 UTC m=+1870.640617029" watchObservedRunningTime="2026-09-09 10:57:08.587082758 +0000 UTC m=+1870.642641808"
Sep 09 10:57:08.709043 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:08.709008    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:57:09.457390 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:09.457353    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:09.462240 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:09.462217    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:09.567145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:09.567120    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:09.568218 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:09.568198    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:11.105759 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.105736    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:57:11.199828 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.199776    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-data\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.199828 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.199842    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istiod-ca-cert\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.200089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.199884    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-envoy\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.200089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.199968    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jj5p\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-kube-api-access-6jj5p\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.200089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200003    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-socket\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.200089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200026    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-token\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.200089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200055    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-podinfo\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.200089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200083    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-credential-socket\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.200403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200155    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-data" (OuterVolumeSpecName: "istio-data") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:57:11.200403 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200181    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-certs\") pod \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\" (UID: \"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e\") "
Sep 09 10:57:11.200510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200372    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:57:11.200510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200398    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:57:11.200510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200462    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:57:11.200510 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200487    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:57:11.200726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200647    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.200726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200670    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.200726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200681    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.200726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200690    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.200726 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.200697    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.202265 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.202235    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 10:57:11.202372 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.202322    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:57:11.202643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.202612    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:57:11.202643 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.202615    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-kube-api-access-6jj5p" (OuterVolumeSpecName: "kube-api-access-6jj5p") pod "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" (UID: "520d15ce-06b6-4da0-92c7-b6fe3b11cf5e"). InnerVolumeSpecName "kube-api-access-6jj5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:57:11.301807 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.301756    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.301807 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.301800    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jj5p\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-kube-api-access-6jj5p\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.301807 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.301810    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.301807 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.301818    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:57:11.580120 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.580030    2586 generic.go:356] "Generic (PLEG): container finished" podID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerID="f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f" exitCode=0
Sep 09 10:57:11.580120 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.580089    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"
Sep 09 10:57:11.580379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.580122    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" event={"ID":"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e","Type":"ContainerDied","Data":"f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f"}
Sep 09 10:57:11.580379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.580194    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k" event={"ID":"520d15ce-06b6-4da0-92c7-b6fe3b11cf5e","Type":"ContainerDied","Data":"53c6b9e0dfd2c915936a6d5a926108756f2f99fb602a7e0d7abf40b847ce87ea"}
Sep 09 10:57:11.580379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.580225    2586 scope.go:117] "RemoveContainer" containerID="f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f"
Sep 09 10:57:11.590280 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.590259    2586 scope.go:117] "RemoveContainer" containerID="ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223"
Sep 09 10:57:11.599272 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.599253    2586 scope.go:117] "RemoveContainer" containerID="f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f"
Sep 09 10:57:11.599607 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:57:11.599584    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f\": container with ID starting with f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f not found: ID does not exist" containerID="f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f"
Sep 09 10:57:11.599689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.599620    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f"} err="failed to get container status \"f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f\": rpc error: code = NotFound desc = could not find container \"f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f\": container with ID starting with f7445ba0f3350f8f69974871d9ae105ffa1ca8e446c8015d1fff09cedf597e2f not found: ID does not exist"
Sep 09 10:57:11.599689 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.599649    2586 scope.go:117] "RemoveContainer" containerID="ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223"
Sep 09 10:57:11.600002 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:57:11.599976    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223\": container with ID starting with ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223 not found: ID does not exist" containerID="ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223"
Sep 09 10:57:11.600089 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.600009    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223"} err="failed to get container status \"ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223\": rpc error: code = NotFound desc = could not find container \"ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223\": container with ID starting with ffa8070f5677e2f61c41d99c46c4b54dfd505586d3d9edfe28aaf62a8c1cc223 not found: ID does not exist"
Sep 09 10:57:11.603065 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.603043    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"]
Sep 09 10:57:11.609380 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.609361    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a03d9e-openshift-default-6b7fc68b48-8f84k"]
Sep 09 10:57:11.667587 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.667556    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"]
Sep 09 10:57:11.668174 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.668152    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy"
Sep 09 10:57:11.668174 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.668173    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy"
Sep 09 10:57:11.668332 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.668193    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy"
Sep 09 10:57:11.668332 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.668199    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy"
Sep 09 10:57:11.668332 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.668267    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy"
Sep 09 10:57:11.668332 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.668280    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6152dd-f966-4d23-9d91-42ab3743b64a" containerName="istio-proxy"
Sep 09 10:57:11.673105 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.673082    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.675848 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.675827    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c1bfbb-serving-cert\""
Sep 09 10:57:11.687490 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.687465    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"]
Sep 09 10:57:11.808145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.808099    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7354b-8015-45f3-b78f-a2fc54a253af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.808145 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.808146    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-metrics-certs\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.808351 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.808205    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8bmj\" (UniqueName: \"kubernetes.io/projected/ebe7354b-8015-45f3-b78f-a2fc54a253af-kube-api-access-h8bmj\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.808351 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.808270    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.909263 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.909225    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7354b-8015-45f3-b78f-a2fc54a253af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.909418 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.909269    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-metrics-certs\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.909418 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.909295    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8bmj\" (UniqueName: \"kubernetes.io/projected/ebe7354b-8015-45f3-b78f-a2fc54a253af-kube-api-access-h8bmj\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.909509 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.909481    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.909624 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:57:11.909607    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-serving-cert: secret "maas-api-e2e-ipp-b-c1bfbb-serving-cert" not found
Sep 09 10:57:11.909709 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:57:11.909694    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls podName:ebe7354b-8015-45f3-b78f-a2fc54a253af nodeName:}" failed. No retries permitted until 2026-09-09 10:57:12.409673517 +0000 UTC m=+1874.465232559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls") pod "maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29" (UID: "ebe7354b-8015-45f3-b78f-a2fc54a253af") : secret "maas-api-e2e-ipp-b-c1bfbb-serving-cert" not found
Sep 09 10:57:11.910042 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.910020    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7354b-8015-45f3-b78f-a2fc54a253af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.912001 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.911977    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-metrics-certs\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:11.919808 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:11.919783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8bmj\" (UniqueName: \"kubernetes.io/projected/ebe7354b-8015-45f3-b78f-a2fc54a253af-kube-api-access-h8bmj\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:12.415214 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:12.415170    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:12.417540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:12.417517    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:12.475631 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:12.475595    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" path="/var/lib/kubelet/pods/520d15ce-06b6-4da0-92c7-b6fe3b11cf5e/volumes"
Sep 09 10:57:12.588218 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:12.588184    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:12.864548 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:12.864519    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"]
Sep 09 10:57:12.866116 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:57:12.866089    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe7354b_8015_45f3_b78f_a2fc54a253af.slice/crio-f65ef89747f43636ff01dd39c2796e8936ec8da19d4bd0b8b89cae6d7722aa81 WatchSource:0}: Error finding container f65ef89747f43636ff01dd39c2796e8936ec8da19d4bd0b8b89cae6d7722aa81: Status 404 returned error can't find the container with id f65ef89747f43636ff01dd39c2796e8936ec8da19d4bd0b8b89cae6d7722aa81
Sep 09 10:57:13.595027 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:13.594989    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29" event={"ID":"ebe7354b-8015-45f3-b78f-a2fc54a253af","Type":"ContainerStarted","Data":"f65ef89747f43636ff01dd39c2796e8936ec8da19d4bd0b8b89cae6d7722aa81"}
Sep 09 10:57:14.603308 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:14.603224    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29" event={"ID":"ebe7354b-8015-45f3-b78f-a2fc54a253af","Type":"ContainerStarted","Data":"c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b"}
Sep 09 10:57:14.603684 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:14.603348    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:14.621545 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:14.621496    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29" podStartSLOduration=2.254477521 podStartE2EDuration="3.621479486s" totalImagesPullingTime="1.367001965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:57:14.620238876 +0000 UTC m=+1876.675797953" watchObservedRunningTime="2026-09-09 10:57:14.621479486 +0000 UTC m=+1876.677038536"
Sep 09 10:57:18.510113 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:18.510079    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:57:20.615211 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:20.615137    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 10:57:24.568644 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:24.568542    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 09 10:57:24.568644 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:24.568602    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 09 10:57:24.683857 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:24.683818    2586 generic.go:356] "Generic (PLEG): container finished" podID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerID="4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8" exitCode=0
Sep 09 10:57:24.684040 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:24.683906    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" event={"ID":"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f","Type":"ContainerDied","Data":"4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8"}
Sep 09 10:57:24.684323 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:24.684305    2586 scope.go:117] "RemoveContainer" containerID="4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8"
Sep 09 10:57:24.685491 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:24.685464    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:24.685545 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:24.685530    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:24.685591 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:24.685561    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 10:57:25.691675 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:25.691639    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" event={"ID":"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f","Type":"ContainerStarted","Data":"f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a"}
Sep 09 10:57:26.456698 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:26.456659    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:26.456928 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:26.456877    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 09 10:57:26.456990 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:26.456963    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 09 10:57:27.457770 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:27.457731    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 09 10:57:27.458204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:27.457789    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 09 10:57:28.456975 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:28.456935    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 09 10:57:28.457186 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:28.457015    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 09 10:57:29.456806 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:29.456772    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 09 10:57:29.457289 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:29.456833    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 09 10:57:30.457479 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:30.457438    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 09 10:57:30.457868 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:30.457499    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 09 10:57:31.461503 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:31.461470    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:31.461985 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:31.461769    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:31.462593 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:31.462577    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 10:57:57.398058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.398024    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"]
Sep 09 10:57:57.398532 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.398524    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="520d15ce-06b6-4da0-92c7-b6fe3b11cf5e" containerName="istio-proxy"
Sep 09 10:57:57.401995 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.401976    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.404361 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.404332    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-33d685\"/\"openshift-service-ca.crt\""
Sep 09 10:57:57.404872 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.404851    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-33d685\"/\"default-dockercfg-gzft4\""
Sep 09 10:57:57.405002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.404906    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-33d685\"/\"kube-root-ca.crt\""
Sep 09 10:57:57.405002 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.404936    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-33d685\"/\"ipp-route-33d685-kserve-self-signed-certs\""
Sep 09 10:57:57.411266 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.411244    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"]
Sep 09 10:57:57.547131 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.547089    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-model-cache\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.547343 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.547152    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tls-certs\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.547343 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.547214    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/f62c3402-e26e-4328-9ec7-5d3ac22c108c-kube-api-access-ncvbq\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.547343 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.547329    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-dshm\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.547492 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.547360    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tmp-dir\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.547492 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.547385    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-home\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.648862 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.648776    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-dshm\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.648862 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.648813    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tmp-dir\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.648862 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.648830    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-home\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.649162 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.648869    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-model-cache\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.649162 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.648922    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tls-certs\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.649162 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.648951    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/f62c3402-e26e-4328-9ec7-5d3ac22c108c-kube-api-access-ncvbq\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.649342 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.649242    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tmp-dir\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.649342 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.649289    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-home\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.649342 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.649327    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-model-cache\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.651045 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.651017    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-dshm\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.651282 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.651263    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tls-certs\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.656608 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.656583    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/f62c3402-e26e-4328-9ec7-5d3ac22c108c-kube-api-access-ncvbq\") pod \"ipp-route-33d685-kserve-59dcbbd6b7-4bf7v\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") " pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.714292 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.714260    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:57:57.987816 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:57.987787    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"]
Sep 09 10:57:57.988967 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:57:57.988932    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf62c3402_e26e_4328_9ec7_5d3ac22c108c.slice/crio-18453c6ac49bf6d687074cd85509dca3c534f1dbec1c32f6d561f1f252d0687f WatchSource:0}: Error finding container 18453c6ac49bf6d687074cd85509dca3c534f1dbec1c32f6d561f1f252d0687f: Status 404 returned error can't find the container with id 18453c6ac49bf6d687074cd85509dca3c534f1dbec1c32f6d561f1f252d0687f
Sep 09 10:57:58.776266 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:58.776228    2586 scope.go:117] "RemoveContainer" containerID="c04e1c9c03ed658cbbe00c56b6d8a3be3ee1ac44b811001be07e14f5a92c2fce"
Sep 09 10:57:58.785300 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:58.785280    2586 scope.go:117] "RemoveContainer" containerID="1676c8fbd5ecf57fe802b3413910c4522954935a471c5f21c46ba05b489255a8"
Sep 09 10:57:58.885966 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:58.885929    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v" event={"ID":"f62c3402-e26e-4328-9ec7-5d3ac22c108c","Type":"ContainerStarted","Data":"dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099"}
Sep 09 10:57:58.886125 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:58.885979    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v" event={"ID":"f62c3402-e26e-4328-9ec7-5d3ac22c108c","Type":"ContainerStarted","Data":"18453c6ac49bf6d687074cd85509dca3c534f1dbec1c32f6d561f1f252d0687f"}
Sep 09 10:57:58.903934 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:57:58.903866    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v" podStartSLOduration=1.903852155 podStartE2EDuration="1.903852155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:57:58.902516938 +0000 UTC m=+1920.958075988" watchObservedRunningTime="2026-09-09 10:57:58.903852155 +0000 UTC m=+1920.959411205"
Sep 09 10:58:01.322204 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:01.322153    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:58:07.714402 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:07.714362    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:58:07.714789 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:07.714420    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:58:07.727067 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:07.727040    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:58:07.960055 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:07.960025    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:58:15.802869 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:15.802832    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:58:27.016909 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.016859    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"]
Sep 09 10:58:27.021369 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.021346    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.023404 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.023380    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-f7ff8b-serving-cert\""
Sep 09 10:58:27.030389 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.030363    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"]
Sep 09 10:58:27.142722 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.142686    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pfv\" (UniqueName: \"kubernetes.io/projected/5c991ae0-ec11-4ba6-98b6-64024e681851-kube-api-access-x4pfv\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.142722 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.142724    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.142982 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.142859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c991ae0-ec11-4ba6-98b6-64024e681851-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.142982 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.142936    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.244147 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.244109    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.244374 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.244163    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pfv\" (UniqueName: \"kubernetes.io/projected/5c991ae0-ec11-4ba6-98b6-64024e681851-kube-api-access-x4pfv\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.244374 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.244284    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.244501 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:58:27.244439    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-serving-cert: secret "maas-api-e2e-ipp-cleanup-f7ff8b-serving-cert" not found
Sep 09 10:58:27.244501 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.244454    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c991ae0-ec11-4ba6-98b6-64024e681851-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.244501 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:58:27.244497    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls podName:5c991ae0-ec11-4ba6-98b6-64024e681851 nodeName:}" failed. No retries permitted until 2026-09-09 10:58:27.744481342 +0000 UTC m=+1949.800040370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf" (UID: "5c991ae0-ec11-4ba6-98b6-64024e681851") : secret "maas-api-e2e-ipp-cleanup-f7ff8b-serving-cert" not found
Sep 09 10:58:27.245100 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.245080    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c991ae0-ec11-4ba6-98b6-64024e681851-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.246740 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.246719    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.254174 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.254146    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pfv\" (UniqueName: \"kubernetes.io/projected/5c991ae0-ec11-4ba6-98b6-64024e681851-kube-api-access-x4pfv\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.751379 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.751342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.753691 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.753672    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:27.936692 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:27.936657    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:28.213251 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:28.213225    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"]
Sep 09 10:58:28.214217 ip-10-0-131-250 kubenswrapper[2586]: W0909 10:58:28.214192    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c991ae0_ec11_4ba6_98b6_64024e681851.slice/crio-0759794c80adc1ebd69eb9d16070d2ecca9964e3c7dedea5dea450f9408fdf7c WatchSource:0}: Error finding container 0759794c80adc1ebd69eb9d16070d2ecca9964e3c7dedea5dea450f9408fdf7c: Status 404 returned error can't find the container with id 0759794c80adc1ebd69eb9d16070d2ecca9964e3c7dedea5dea450f9408fdf7c
Sep 09 10:58:29.078823 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:29.078732    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf" event={"ID":"5c991ae0-ec11-4ba6-98b6-64024e681851","Type":"ContainerStarted","Data":"0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931"}
Sep 09 10:58:29.078823 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:29.078787    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf" event={"ID":"5c991ae0-ec11-4ba6-98b6-64024e681851","Type":"ContainerStarted","Data":"0759794c80adc1ebd69eb9d16070d2ecca9964e3c7dedea5dea450f9408fdf7c"}
Sep 09 10:58:29.079077 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:29.078887    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:29.099568 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:29.099520    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf" podStartSLOduration=2.529227358 podStartE2EDuration="3.099504677s" totalImagesPullingTime="570.277319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:58:29.09649348 +0000 UTC m=+1951.152052531" watchObservedRunningTime="2026-09-09 10:58:29.099504677 +0000 UTC m=+1951.155063726"
Sep 09 10:58:35.095130 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:35.095096    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:35.511317 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:35.511282    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:58:57.704344 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:57.704227    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"]
Sep 09 10:58:57.704882 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:57.704491    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf" podUID="5c991ae0-ec11-4ba6-98b6-64024e681851" containerName="maas-api" containerID="cri-o://0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931" gracePeriod=30
Sep 09 10:58:57.954175 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:57.954150    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:58.054631 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.054552    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4pfv\" (UniqueName: \"kubernetes.io/projected/5c991ae0-ec11-4ba6-98b6-64024e681851-kube-api-access-x4pfv\") pod \"5c991ae0-ec11-4ba6-98b6-64024e681851\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") "
Sep 09 10:58:58.054782 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.054665    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls\") pod \"5c991ae0-ec11-4ba6-98b6-64024e681851\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") "
Sep 09 10:58:58.054782 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.054715    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c991ae0-ec11-4ba6-98b6-64024e681851-openshift-service-ca\") pod \"5c991ae0-ec11-4ba6-98b6-64024e681851\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") "
Sep 09 10:58:58.054782 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.054744    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-metrics-certs\") pod \"5c991ae0-ec11-4ba6-98b6-64024e681851\" (UID: \"5c991ae0-ec11-4ba6-98b6-64024e681851\") "
Sep 09 10:58:58.055130 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.055101    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c991ae0-ec11-4ba6-98b6-64024e681851-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c991ae0-ec11-4ba6-98b6-64024e681851" (UID: "5c991ae0-ec11-4ba6-98b6-64024e681851"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 10:58:58.056860 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.056828    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c991ae0-ec11-4ba6-98b6-64024e681851" (UID: "5c991ae0-ec11-4ba6-98b6-64024e681851"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:58:58.056860 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.056830    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c991ae0-ec11-4ba6-98b6-64024e681851-kube-api-access-x4pfv" (OuterVolumeSpecName: "kube-api-access-x4pfv") pod "5c991ae0-ec11-4ba6-98b6-64024e681851" (UID: "5c991ae0-ec11-4ba6-98b6-64024e681851"). InnerVolumeSpecName "kube-api-access-x4pfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:58:58.057040 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.056857    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c991ae0-ec11-4ba6-98b6-64024e681851" (UID: "5c991ae0-ec11-4ba6-98b6-64024e681851"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:58:58.156331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.156297    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c991ae0-ec11-4ba6-98b6-64024e681851-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:58:58.156331 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.156326    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:58:58.156540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.156335    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pfv\" (UniqueName: \"kubernetes.io/projected/5c991ae0-ec11-4ba6-98b6-64024e681851-kube-api-access-x4pfv\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:58:58.156540 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.156356    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c991ae0-ec11-4ba6-98b6-64024e681851-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:58:58.262424 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.262391    2586 generic.go:356] "Generic (PLEG): container finished" podID="5c991ae0-ec11-4ba6-98b6-64024e681851" containerID="0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931" exitCode=0
Sep 09 10:58:58.262592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.262447    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"
Sep 09 10:58:58.262592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.262478    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf" event={"ID":"5c991ae0-ec11-4ba6-98b6-64024e681851","Type":"ContainerDied","Data":"0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931"}
Sep 09 10:58:58.262592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.262532    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf" event={"ID":"5c991ae0-ec11-4ba6-98b6-64024e681851","Type":"ContainerDied","Data":"0759794c80adc1ebd69eb9d16070d2ecca9964e3c7dedea5dea450f9408fdf7c"}
Sep 09 10:58:58.262592 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.262554    2586 scope.go:117] "RemoveContainer" containerID="0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931"
Sep 09 10:58:58.272573 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.272558    2586 scope.go:117] "RemoveContainer" containerID="0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931"
Sep 09 10:58:58.272871 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:58:58.272850    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931\": container with ID starting with 0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931 not found: ID does not exist" containerID="0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931"
Sep 09 10:58:58.273009 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.272881    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931"} err="failed to get container status \"0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931\": rpc error: code = NotFound desc = could not find container \"0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931\": container with ID starting with 0943c50e593852ac800f4332388c1c417a42c848bd4ae4999f8924db84bd1931 not found: ID does not exist"
Sep 09 10:58:58.287199 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.287172    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"]
Sep 09 10:58:58.297332 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.297306    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f7ff8b-75c8866c9b-kxftf"]
Sep 09 10:58:58.476292 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.476262    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c991ae0-ec11-4ba6-98b6-64024e681851" path="/var/lib/kubelet/pods/5c991ae0-ec11-4ba6-98b6-64024e681851/volumes"
Sep 09 10:58:58.815537 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:58:58.815457    2586 scope.go:117] "RemoveContainer" containerID="693f4d9110f53fbbdb97b6298e5d039dfd66edddd1dbc493b87903dacffdbd00"
Sep 09 10:59:02.208806 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:02.208765    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:59:22.610661 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:22.610627    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:59:30.706040 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:30.705999    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"]
Sep 09 10:59:34.508700 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:34.508658    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 10:59:45.726293 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:45.726222    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v" podUID="f62c3402-e26e-4328-9ec7-5d3ac22c108c" containerName="main" containerID="cri-o://dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099" gracePeriod=45
Sep 09 10:59:45.971845 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:45.971817    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:59:46.131994 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.131954    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-home\") pod \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") "
Sep 09 10:59:46.132165 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132033    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tmp-dir\") pod \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") "
Sep 09 10:59:46.132165 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132052    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-dshm\") pod \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") "
Sep 09 10:59:46.132165 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132133    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-model-cache\") pod \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") "
Sep 09 10:59:46.132273 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132196    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/f62c3402-e26e-4328-9ec7-5d3ac22c108c-kube-api-access-ncvbq\") pod \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") "
Sep 09 10:59:46.132273 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132218    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tls-certs\") pod \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\" (UID: \"f62c3402-e26e-4328-9ec7-5d3ac22c108c\") "
Sep 09 10:59:46.132419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132315    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-home" (OuterVolumeSpecName: "home") pod "f62c3402-e26e-4328-9ec7-5d3ac22c108c" (UID: "f62c3402-e26e-4328-9ec7-5d3ac22c108c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:46.132419 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132324    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f62c3402-e26e-4328-9ec7-5d3ac22c108c" (UID: "f62c3402-e26e-4328-9ec7-5d3ac22c108c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:46.132529 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132507    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f62c3402-e26e-4328-9ec7-5d3ac22c108c" (UID: "f62c3402-e26e-4328-9ec7-5d3ac22c108c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:46.132576 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132559    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:59:46.132576 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.132571    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:59:46.134544 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.134509    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f62c3402-e26e-4328-9ec7-5d3ac22c108c" (UID: "f62c3402-e26e-4328-9ec7-5d3ac22c108c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 10:59:46.134664 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.134567    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-dshm" (OuterVolumeSpecName: "dshm") pod "f62c3402-e26e-4328-9ec7-5d3ac22c108c" (UID: "f62c3402-e26e-4328-9ec7-5d3ac22c108c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 10:59:46.134664 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.134580    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62c3402-e26e-4328-9ec7-5d3ac22c108c-kube-api-access-ncvbq" (OuterVolumeSpecName: "kube-api-access-ncvbq") pod "f62c3402-e26e-4328-9ec7-5d3ac22c108c" (UID: "f62c3402-e26e-4328-9ec7-5d3ac22c108c"). InnerVolumeSpecName "kube-api-access-ncvbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 10:59:46.234022 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.233982    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:59:46.234197 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.234039    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/f62c3402-e26e-4328-9ec7-5d3ac22c108c-kube-api-access-ncvbq\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:59:46.234197 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.234056    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62c3402-e26e-4328-9ec7-5d3ac22c108c-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:59:46.234197 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.234068    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62c3402-e26e-4328-9ec7-5d3ac22c108c-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 10:59:46.554864 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.554830    2586 generic.go:356] "Generic (PLEG): container finished" podID="f62c3402-e26e-4328-9ec7-5d3ac22c108c" containerID="dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099" exitCode=0
Sep 09 10:59:46.555058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.554872    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"
Sep 09 10:59:46.555058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.554932    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v" event={"ID":"f62c3402-e26e-4328-9ec7-5d3ac22c108c","Type":"ContainerDied","Data":"dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099"}
Sep 09 10:59:46.555058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.554978    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v" event={"ID":"f62c3402-e26e-4328-9ec7-5d3ac22c108c","Type":"ContainerDied","Data":"18453c6ac49bf6d687074cd85509dca3c534f1dbec1c32f6d561f1f252d0687f"}
Sep 09 10:59:46.555058 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.554998    2586 scope.go:117] "RemoveContainer" containerID="dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099"
Sep 09 10:59:46.564514 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.564499    2586 scope.go:117] "RemoveContainer" containerID="dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099"
Sep 09 10:59:46.564832 ip-10-0-131-250 kubenswrapper[2586]: E0909 10:59:46.564813    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099\": container with ID starting with dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099 not found: ID does not exist" containerID="dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099"
Sep 09 10:59:46.564995 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.564844    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099"} err="failed to get container status \"dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099\": rpc error: code = NotFound desc = could not find container \"dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099\": container with ID starting with dd9a8696cfd689d5ffcafc3538722c2093a34066d98dc205b0b9566ff4cec099 not found: ID does not exist"
Sep 09 10:59:46.572645 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.572615    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"]
Sep 09 10:59:46.576780 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:46.576756    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-33d685/ipp-route-33d685-kserve-59dcbbd6b7-4bf7v"]
Sep 09 10:59:48.475455 ip-10-0-131-250 kubenswrapper[2586]: I0909 10:59:48.475420    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f62c3402-e26e-4328-9ec7-5d3ac22c108c" path="/var/lib/kubelet/pods/f62c3402-e26e-4328-9ec7-5d3ac22c108c/volumes"
Sep 09 11:00:00.146281 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.146241    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"]
Sep 09 11:00:00.146783 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.146768    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c991ae0-ec11-4ba6-98b6-64024e681851" containerName="maas-api"
Sep 09 11:00:00.146839 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.146786    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c991ae0-ec11-4ba6-98b6-64024e681851" containerName="maas-api"
Sep 09 11:00:00.146839 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.146795    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62c3402-e26e-4328-9ec7-5d3ac22c108c" containerName="main"
Sep 09 11:00:00.146839 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.146801    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62c3402-e26e-4328-9ec7-5d3ac22c108c" containerName="main"
Sep 09 11:00:00.146975 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.146875    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62c3402-e26e-4328-9ec7-5d3ac22c108c" containerName="main"
Sep 09 11:00:00.146975 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.146887    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c991ae0-ec11-4ba6-98b6-64024e681851" containerName="maas-api"
Sep 09 11:00:00.151389 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.151372    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"
Sep 09 11:00:00.153782 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.153763    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\""
Sep 09 11:00:00.161366 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.161341    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"]
Sep 09 11:00:00.163480 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.163456    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"
Sep 09 11:00:00.437864 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.437833    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"]
Sep 09 11:00:00.439590 ip-10-0-131-250 kubenswrapper[2586]: W0909 11:00:00.439561    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f35f3a8_f6b1_4445_850b_64efa844baa8.slice/crio-da65abaf937bcb00612af530f53cb38a7a3868ea31f4aedd6d0be64b55232039 WatchSource:0}: Error finding container da65abaf937bcb00612af530f53cb38a7a3868ea31f4aedd6d0be64b55232039: Status 404 returned error can't find the container with id da65abaf937bcb00612af530f53cb38a7a3868ea31f4aedd6d0be64b55232039
Sep 09 11:00:00.641342 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:00.641308    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt" event={"ID":"0f35f3a8-f6b1-4445-850b-64efa844baa8","Type":"ContainerStarted","Data":"da65abaf937bcb00612af530f53cb38a7a3868ea31f4aedd6d0be64b55232039"}
Sep 09 11:00:01.649191 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:01.649150    2586 generic.go:356] "Generic (PLEG): container finished" podID="0f35f3a8-f6b1-4445-850b-64efa844baa8" containerID="e8bb3e5207aa7da16261da7f7cd5fd3babb8888567caf9296ce1a17ac70de6f9" exitCode=0
Sep 09 11:00:01.649578 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:01.649193    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt" event={"ID":"0f35f3a8-f6b1-4445-850b-64efa844baa8","Type":"ContainerDied","Data":"e8bb3e5207aa7da16261da7f7cd5fd3babb8888567caf9296ce1a17ac70de6f9"}
Sep 09 11:00:02.787296 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:02.787266    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"
Sep 09 11:00:03.662578 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:03.662545    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt" event={"ID":"0f35f3a8-f6b1-4445-850b-64efa844baa8","Type":"ContainerDied","Data":"da65abaf937bcb00612af530f53cb38a7a3868ea31f4aedd6d0be64b55232039"}
Sep 09 11:00:03.662790 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:03.662604    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da65abaf937bcb00612af530f53cb38a7a3868ea31f4aedd6d0be64b55232039"
Sep 09 11:00:03.662790 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:03.662555    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"
Sep 09 11:00:25.705350 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:25.705270    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"]
Sep 09 11:00:25.705747 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:25.705567    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29" podUID="ebe7354b-8015-45f3-b78f-a2fc54a253af" containerName="maas-api" containerID="cri-o://c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b" gracePeriod=30
Sep 09 11:00:25.959262 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:25.959209    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 11:00:26.112517 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.112478    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8bmj\" (UniqueName: \"kubernetes.io/projected/ebe7354b-8015-45f3-b78f-a2fc54a253af-kube-api-access-h8bmj\") pod \"ebe7354b-8015-45f3-b78f-a2fc54a253af\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") "
Sep 09 11:00:26.113056 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.112530    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-metrics-certs\") pod \"ebe7354b-8015-45f3-b78f-a2fc54a253af\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") "
Sep 09 11:00:26.113056 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.112586    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls\") pod \"ebe7354b-8015-45f3-b78f-a2fc54a253af\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") "
Sep 09 11:00:26.113056 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.112615    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7354b-8015-45f3-b78f-a2fc54a253af-openshift-service-ca\") pod \"ebe7354b-8015-45f3-b78f-a2fc54a253af\" (UID: \"ebe7354b-8015-45f3-b78f-a2fc54a253af\") "
Sep 09 11:00:26.113285 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.113076    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe7354b-8015-45f3-b78f-a2fc54a253af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebe7354b-8015-45f3-b78f-a2fc54a253af" (UID: "ebe7354b-8015-45f3-b78f-a2fc54a253af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:00:26.115122 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.115095    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe7354b-8015-45f3-b78f-a2fc54a253af-kube-api-access-h8bmj" (OuterVolumeSpecName: "kube-api-access-h8bmj") pod "ebe7354b-8015-45f3-b78f-a2fc54a253af" (UID: "ebe7354b-8015-45f3-b78f-a2fc54a253af"). InnerVolumeSpecName "kube-api-access-h8bmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:00:26.115294 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.115269    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebe7354b-8015-45f3-b78f-a2fc54a253af" (UID: "ebe7354b-8015-45f3-b78f-a2fc54a253af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:00:26.115529 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.115505    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebe7354b-8015-45f3-b78f-a2fc54a253af" (UID: "ebe7354b-8015-45f3-b78f-a2fc54a253af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:00:26.213582 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.213496    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8bmj\" (UniqueName: \"kubernetes.io/projected/ebe7354b-8015-45f3-b78f-a2fc54a253af-kube-api-access-h8bmj\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:26.213582 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.213526    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:26.213582 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.213535    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7354b-8015-45f3-b78f-a2fc54a253af-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:26.213582 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.213544    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7354b-8015-45f3-b78f-a2fc54a253af-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:26.805293 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.805204    2586 generic.go:356] "Generic (PLEG): container finished" podID="ebe7354b-8015-45f3-b78f-a2fc54a253af" containerID="c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b" exitCode=0
Sep 09 11:00:26.805293 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.805262    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"
Sep 09 11:00:26.805748 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.805292    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29" event={"ID":"ebe7354b-8015-45f3-b78f-a2fc54a253af","Type":"ContainerDied","Data":"c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b"}
Sep 09 11:00:26.805748 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.805351    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29" event={"ID":"ebe7354b-8015-45f3-b78f-a2fc54a253af","Type":"ContainerDied","Data":"f65ef89747f43636ff01dd39c2796e8936ec8da19d4bd0b8b89cae6d7722aa81"}
Sep 09 11:00:26.805748 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.805374    2586 scope.go:117] "RemoveContainer" containerID="c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b"
Sep 09 11:00:26.814643 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.814628    2586 scope.go:117] "RemoveContainer" containerID="c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b"
Sep 09 11:00:26.814945 ip-10-0-131-250 kubenswrapper[2586]: E0909 11:00:26.814922    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b\": container with ID starting with c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b not found: ID does not exist" containerID="c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b"
Sep 09 11:00:26.815022 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.814959    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b"} err="failed to get container status \"c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b\": rpc error: code = NotFound desc = could not find container \"c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b\": container with ID starting with c6cd968842e0c7cce25ea1b5c0c15c8f776e9757b7275a2ef9dff1504296f41b not found: ID does not exist"
Sep 09 11:00:26.822900 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.822864    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"]
Sep 09 11:00:26.826599 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:26.826577    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1bfbb-659695b997-hnc29"]
Sep 09 11:00:28.477466 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:28.477431    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe7354b-8015-45f3-b78f-a2fc54a253af" path="/var/lib/kubelet/pods/ebe7354b-8015-45f3-b78f-a2fc54a253af/volumes"
Sep 09 11:00:29.008443 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:29.008402    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:00:32.994194 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:32.994156    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"]
Sep 09 11:00:32.994632 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:32.994387    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy" containerID="cri-o://f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a" gracePeriod=30
Sep 09 11:00:38.233551 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.233518    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 11:00:38.328396 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328310    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-credential-socket\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328396 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328353    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-certs\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328396 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328375    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-socket\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328396 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328395    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-envoy\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328716 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328432    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-podinfo\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328716 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328458    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istiod-ca-cert\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328716 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328526    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jlkm\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-kube-api-access-6jlkm\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328716 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328589    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-data\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328716 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328612    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-token\") pod \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\" (UID: \"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f\") "
Sep 09 11:00:38.328716 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328656    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:38.328716 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328685    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:38.329090 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328912    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:38.329090 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328931    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:38.329090 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328945    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:00:38.329090 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.328984    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.329090 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.329004    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.330808 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.330783    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:00:38.330974 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.330780    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 11:00:38.330974 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.330785    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:00:38.330974 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.330813    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-kube-api-access-6jlkm" (OuterVolumeSpecName: "kube-api-access-6jlkm") pod "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" (UID: "75bc31fa-0ca4-40a8-8c42-fe02414c2c3f"). InnerVolumeSpecName "kube-api-access-6jlkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:00:38.429471 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.429434    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlkm\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-kube-api-access-6jlkm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.429471 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.429465    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.429471 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.429474    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.429471 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.429482    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.429743 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.429490    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.429743 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.429500    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.429743 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.429508    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:38.875355 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.875322    2586 generic.go:356] "Generic (PLEG): container finished" podID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerID="f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a" exitCode=0
Sep 09 11:00:38.875527 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.875366    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"
Sep 09 11:00:38.875527 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.875412    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" event={"ID":"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f","Type":"ContainerDied","Data":"f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a"}
Sep 09 11:00:38.875527 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.875470    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl" event={"ID":"75bc31fa-0ca4-40a8-8c42-fe02414c2c3f","Type":"ContainerDied","Data":"272999905a8127a6f4f54295189300cdccb89ad53be9cc8f382d0a6157b0276b"}
Sep 09 11:00:38.875527 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.875497    2586 scope.go:117] "RemoveContainer" containerID="f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a"
Sep 09 11:00:38.884984 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.884966    2586 scope.go:117] "RemoveContainer" containerID="4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8"
Sep 09 11:00:38.894267 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.894249    2586 scope.go:117] "RemoveContainer" containerID="f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a"
Sep 09 11:00:38.894541 ip-10-0-131-250 kubenswrapper[2586]: E0909 11:00:38.894519    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a\": container with ID starting with f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a not found: ID does not exist" containerID="f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a"
Sep 09 11:00:38.894588 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.894551    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a"} err="failed to get container status \"f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a\": rpc error: code = NotFound desc = could not find container \"f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a\": container with ID starting with f43af1bfa7aaadffe23951994f139c8532a8815b9802778b49627b9f80428c1a not found: ID does not exist"
Sep 09 11:00:38.894588 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.894569    2586 scope.go:117] "RemoveContainer" containerID="4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8"
Sep 09 11:00:38.894852 ip-10-0-131-250 kubenswrapper[2586]: E0909 11:00:38.894828    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8\": container with ID starting with 4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8 not found: ID does not exist" containerID="4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8"
Sep 09 11:00:38.894905 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.894856    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8"} err="failed to get container status \"4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8\": rpc error: code = NotFound desc = could not find container \"4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8\": container with ID starting with 4c62b0d70075b3c51c56a73a1132db088c16afeabd45c472eff3a15efa4c2ae8 not found: ID does not exist"
Sep 09 11:00:38.895569 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.895545    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"]
Sep 09 11:00:38.898560 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:38.898540    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1bfbb-openshift-default-6776b6b86d-wfgwl"]
Sep 09 11:00:40.475261 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:40.475231    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" path="/var/lib/kubelet/pods/75bc31fa-0ca4-40a8-8c42-fe02414c2c3f/volumes"
Sep 09 11:00:43.849251 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:43.849214    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"]
Sep 09 11:00:43.849672 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:43.849473    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9" podUID="2b777b12-6493-4377-9ef3-7617c69381a1" containerName="maas-api" containerID="cri-o://073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3" gracePeriod=30
Sep 09 11:00:43.883618 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:43.883587    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"]
Sep 09 11:00:43.886277 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:43.886256    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1eb21d-29815860-86bnt"]
Sep 09 11:00:44.104379 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.104313    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 11:00:44.181293 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.181264    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j722l\" (UniqueName: \"kubernetes.io/projected/2b777b12-6493-4377-9ef3-7617c69381a1-kube-api-access-j722l\") pod \"2b777b12-6493-4377-9ef3-7617c69381a1\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") "
Sep 09 11:00:44.181293 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.181299    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls\") pod \"2b777b12-6493-4377-9ef3-7617c69381a1\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") "
Sep 09 11:00:44.181503 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.181328    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b777b12-6493-4377-9ef3-7617c69381a1-openshift-service-ca\") pod \"2b777b12-6493-4377-9ef3-7617c69381a1\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") "
Sep 09 11:00:44.181503 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.181415    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-metrics-certs\") pod \"2b777b12-6493-4377-9ef3-7617c69381a1\" (UID: \"2b777b12-6493-4377-9ef3-7617c69381a1\") "
Sep 09 11:00:44.181766 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.181737    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b777b12-6493-4377-9ef3-7617c69381a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b777b12-6493-4377-9ef3-7617c69381a1" (UID: "2b777b12-6493-4377-9ef3-7617c69381a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:00:44.183536 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.183503    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b777b12-6493-4377-9ef3-7617c69381a1" (UID: "2b777b12-6493-4377-9ef3-7617c69381a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:00:44.183636 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.183555    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b777b12-6493-4377-9ef3-7617c69381a1" (UID: "2b777b12-6493-4377-9ef3-7617c69381a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:00:44.183636 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.183567    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b777b12-6493-4377-9ef3-7617c69381a1-kube-api-access-j722l" (OuterVolumeSpecName: "kube-api-access-j722l") pod "2b777b12-6493-4377-9ef3-7617c69381a1" (UID: "2b777b12-6493-4377-9ef3-7617c69381a1"). InnerVolumeSpecName "kube-api-access-j722l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:00:44.282748 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.282714    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j722l\" (UniqueName: \"kubernetes.io/projected/2b777b12-6493-4377-9ef3-7617c69381a1-kube-api-access-j722l\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:44.282748 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.282742    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:44.282748 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.282751    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b777b12-6493-4377-9ef3-7617c69381a1-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:44.283033 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.282759    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b777b12-6493-4377-9ef3-7617c69381a1-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:00:44.474798 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.474764    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f35f3a8-f6b1-4445-850b-64efa844baa8" path="/var/lib/kubelet/pods/0f35f3a8-f6b1-4445-850b-64efa844baa8/volumes"
Sep 09 11:00:44.916884 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.916847    2586 generic.go:356] "Generic (PLEG): container finished" podID="2b777b12-6493-4377-9ef3-7617c69381a1" containerID="073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3" exitCode=0
Sep 09 11:00:44.917343 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.916922    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"
Sep 09 11:00:44.917343 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.916935    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9" event={"ID":"2b777b12-6493-4377-9ef3-7617c69381a1","Type":"ContainerDied","Data":"073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3"}
Sep 09 11:00:44.917343 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.916990    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9" event={"ID":"2b777b12-6493-4377-9ef3-7617c69381a1","Type":"ContainerDied","Data":"61a07c6d06138f6d7ca868ce714fe8936fc4bef8fc6187699c27959c7837802e"}
Sep 09 11:00:44.917343 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.917013    2586 scope.go:117] "RemoveContainer" containerID="073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3"
Sep 09 11:00:44.926625 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.926599    2586 scope.go:117] "RemoveContainer" containerID="073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3"
Sep 09 11:00:44.926926 ip-10-0-131-250 kubenswrapper[2586]: E0909 11:00:44.926877    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3\": container with ID starting with 073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3 not found: ID does not exist" containerID="073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3"
Sep 09 11:00:44.927001 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.926921    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3"} err="failed to get container status \"073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3\": rpc error: code = NotFound desc = could not find container \"073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3\": container with ID starting with 073d6bb8e6a54a14bcb5df57fd80f4293ce34759ad64a04841a4d68c54719be3 not found: ID does not exist"
Sep 09 11:00:44.933047 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.933024    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"]
Sep 09 11:00:44.937808 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:44.937788    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1eb21d-58cd64858d-mq8w9"]
Sep 09 11:00:46.475485 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:46.475453    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b777b12-6493-4377-9ef3-7617c69381a1" path="/var/lib/kubelet/pods/2b777b12-6493-4377-9ef3-7617c69381a1/volumes"
Sep 09 11:00:46.913183 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:46.913144    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:00:58.856529 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:58.856493    2586 scope.go:117] "RemoveContainer" containerID="12ffd77c93087ddb4f05f7648708c7f81791729d7699681b0b3e5860f0bd9f0a"
Sep 09 11:00:58.865507 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:00:58.865488    2586 scope.go:117] "RemoveContainer" containerID="e2e3482491ca2c2d968d2a8e16c6983f686a4a740189e2254f555390d3b64d61"
Sep 09 11:01:04.803940 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:01:04.803902    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:01:14.607714 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:01:14.607678    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:02:39.503559 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:02:39.503521    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:03:07.100951 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:03:07.100911    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:03:15.003356 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:03:15.003318    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:03:24.210187 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:03:24.210109    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:03:39.291745 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:03:39.291711    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:03:44.920864 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:03:44.920822    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:05:15.214832 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:15.214747    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:05:20.954776 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:20.954742    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:05:31.605401 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:31.605362    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:05:33.505842 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.505802    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"]
Sep 09 11:05:33.506392 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506372    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy"
Sep 09 11:05:33.506487 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506394    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy"
Sep 09 11:05:33.506487 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506414    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy"
Sep 09 11:05:33.506487 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506424    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy"
Sep 09 11:05:33.506487 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506435    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b777b12-6493-4377-9ef3-7617c69381a1" containerName="maas-api"
Sep 09 11:05:33.506487 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506444    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b777b12-6493-4377-9ef3-7617c69381a1" containerName="maas-api"
Sep 09 11:05:33.506487 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506469    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe7354b-8015-45f3-b78f-a2fc54a253af" containerName="maas-api"
Sep 09 11:05:33.506487 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506477    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe7354b-8015-45f3-b78f-a2fc54a253af" containerName="maas-api"
Sep 09 11:05:33.506826 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506499    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f35f3a8-f6b1-4445-850b-64efa844baa8" containerName="cleanup"
Sep 09 11:05:33.506826 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506509    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f35f3a8-f6b1-4445-850b-64efa844baa8" containerName="cleanup"
Sep 09 11:05:33.506826 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506609    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe7354b-8015-45f3-b78f-a2fc54a253af" containerName="maas-api"
Sep 09 11:05:33.506826 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506623    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy"
Sep 09 11:05:33.506826 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506635    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b777b12-6493-4377-9ef3-7617c69381a1" containerName="maas-api"
Sep 09 11:05:33.506826 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.506644    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f35f3a8-f6b1-4445-850b-64efa844baa8" containerName="cleanup"
Sep 09 11:05:33.509980 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.509959    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.512440 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.512410    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fe318cb6-kserve-self-signed-certs\""
Sep 09 11:05:33.525271 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.525243    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"]
Sep 09 11:05:33.649818 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.649775    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-dshm\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.649818 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.649825    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f020a1-4498-4171-8334-969f6d19fed8-tls-certs\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.650061 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.649856    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-model-cache\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.650061 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.649878    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btnj\" (UniqueName: \"kubernetes.io/projected/10f020a1-4498-4171-8334-969f6d19fed8-kube-api-access-5btnj\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.650061 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.650001    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-home\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.650162 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.650059    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-tmp-dir\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.751512 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.751477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-tmp-dir\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.751672 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.751531    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-dshm\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.751672 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.751642    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f020a1-4498-4171-8334-969f6d19fed8-tls-certs\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.751780 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.751684    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-model-cache\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.751780 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.751701    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btnj\" (UniqueName: \"kubernetes.io/projected/10f020a1-4498-4171-8334-969f6d19fed8-kube-api-access-5btnj\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.751918 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.751869    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-tmp-dir\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.751990 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.751939    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-home\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.752047 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.752022    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-model-cache\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.752229 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.752212    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-home\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.753921 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.753868    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-dshm\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.754162 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.754147    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f020a1-4498-4171-8334-969f6d19fed8-tls-certs\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.761929 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.761866    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btnj\" (UniqueName: \"kubernetes.io/projected/10f020a1-4498-4171-8334-969f6d19fed8-kube-api-access-5btnj\") pod \"e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") " pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:33.819546 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:33.819518    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:34.110137 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:34.110115    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"]
Sep 09 11:05:34.111772 ip-10-0-131-250 kubenswrapper[2586]: W0909 11:05:34.111741    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10f020a1_4498_4171_8334_969f6d19fed8.slice/crio-8a184030ca1ab4ccb70fd0fb1260da520636182cb2d9e528fe7e357923b7b953 WatchSource:0}: Error finding container 8a184030ca1ab4ccb70fd0fb1260da520636182cb2d9e528fe7e357923b7b953: Status 404 returned error can't find the container with id 8a184030ca1ab4ccb70fd0fb1260da520636182cb2d9e528fe7e357923b7b953
Sep 09 11:05:34.559747 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:34.559709    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q" event={"ID":"10f020a1-4498-4171-8334-969f6d19fed8","Type":"ContainerStarted","Data":"46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1"}
Sep 09 11:05:34.559747 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:34.559753    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q" event={"ID":"10f020a1-4498-4171-8334-969f6d19fed8","Type":"ContainerStarted","Data":"8a184030ca1ab4ccb70fd0fb1260da520636182cb2d9e528fe7e357923b7b953"}
Sep 09 11:05:34.604654 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:34.604607    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q" podStartSLOduration=1.6045914049999999 podStartE2EDuration="1.604591405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:05:34.584696122 +0000 UTC m=+2376.640255172" watchObservedRunningTime="2026-09-09 11:05:34.604591405 +0000 UTC m=+2376.660150454"
Sep 09 11:05:42.704934 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:42.704901    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:05:43.820535 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:43.820495    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:43.820946 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:43.820579    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:43.833197 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:43.833169    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:44.627452 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:44.627424    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:05:49.781781 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:49.781742    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"]
Sep 09 11:05:49.924035 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:49.923994    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:05:59.113421 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:05:59.113388    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:06:04.800993 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:04.800930    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q" podUID="10f020a1-4498-4171-8334-969f6d19fed8" containerName="main" containerID="cri-o://46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1" gracePeriod=45
Sep 09 11:06:05.052500 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.052438    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:06:05.155952 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.155882    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-home\") pod \"10f020a1-4498-4171-8334-969f6d19fed8\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") "
Sep 09 11:06:05.155952 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.155956    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-tmp-dir\") pod \"10f020a1-4498-4171-8334-969f6d19fed8\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") "
Sep 09 11:06:05.156187 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.155984    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-dshm\") pod \"10f020a1-4498-4171-8334-969f6d19fed8\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") "
Sep 09 11:06:05.156187 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.156070    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-model-cache\") pod \"10f020a1-4498-4171-8334-969f6d19fed8\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") "
Sep 09 11:06:05.156187 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.156105    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f020a1-4498-4171-8334-969f6d19fed8-tls-certs\") pod \"10f020a1-4498-4171-8334-969f6d19fed8\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") "
Sep 09 11:06:05.156187 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.156120    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5btnj\" (UniqueName: \"kubernetes.io/projected/10f020a1-4498-4171-8334-969f6d19fed8-kube-api-access-5btnj\") pod \"10f020a1-4498-4171-8334-969f6d19fed8\" (UID: \"10f020a1-4498-4171-8334-969f6d19fed8\") "
Sep 09 11:06:05.156404 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.156357    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10f020a1-4498-4171-8334-969f6d19fed8" (UID: "10f020a1-4498-4171-8334-969f6d19fed8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:06:05.156464 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.156351    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-home" (OuterVolumeSpecName: "home") pod "10f020a1-4498-4171-8334-969f6d19fed8" (UID: "10f020a1-4498-4171-8334-969f6d19fed8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:06:05.156464 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.156426    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-model-cache" (OuterVolumeSpecName: "model-cache") pod "10f020a1-4498-4171-8334-969f6d19fed8" (UID: "10f020a1-4498-4171-8334-969f6d19fed8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:06:05.158310 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.158281    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-dshm" (OuterVolumeSpecName: "dshm") pod "10f020a1-4498-4171-8334-969f6d19fed8" (UID: "10f020a1-4498-4171-8334-969f6d19fed8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 11:06:05.158423 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.158316    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f020a1-4498-4171-8334-969f6d19fed8-kube-api-access-5btnj" (OuterVolumeSpecName: "kube-api-access-5btnj") pod "10f020a1-4498-4171-8334-969f6d19fed8" (UID: "10f020a1-4498-4171-8334-969f6d19fed8"). InnerVolumeSpecName "kube-api-access-5btnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:06:05.158423 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.158341    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f020a1-4498-4171-8334-969f6d19fed8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10f020a1-4498-4171-8334-969f6d19fed8" (UID: "10f020a1-4498-4171-8334-969f6d19fed8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:06:05.257829 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.257793    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:05.257829 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.257825    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f020a1-4498-4171-8334-969f6d19fed8-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:05.257829 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.257834    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5btnj\" (UniqueName: \"kubernetes.io/projected/10f020a1-4498-4171-8334-969f6d19fed8-kube-api-access-5btnj\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:05.258093 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.257844    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:05.258093 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.257853    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:05.258093 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.257865    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f020a1-4498-4171-8334-969f6d19fed8-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:05.738225 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.738186    2586 generic.go:356] "Generic (PLEG): container finished" podID="10f020a1-4498-4171-8334-969f6d19fed8" containerID="46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1" exitCode=0
Sep 09 11:06:05.738406 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.738234    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"
Sep 09 11:06:05.738406 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.738271    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q" event={"ID":"10f020a1-4498-4171-8334-969f6d19fed8","Type":"ContainerDied","Data":"46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1"}
Sep 09 11:06:05.738406 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.738322    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q" event={"ID":"10f020a1-4498-4171-8334-969f6d19fed8","Type":"ContainerDied","Data":"8a184030ca1ab4ccb70fd0fb1260da520636182cb2d9e528fe7e357923b7b953"}
Sep 09 11:06:05.738406 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.738341    2586 scope.go:117] "RemoveContainer" containerID="46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1"
Sep 09 11:06:05.748536 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.748515    2586 scope.go:117] "RemoveContainer" containerID="46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1"
Sep 09 11:06:05.748811 ip-10-0-131-250 kubenswrapper[2586]: E0909 11:06:05.748789    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1\": container with ID starting with 46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1 not found: ID does not exist" containerID="46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1"
Sep 09 11:06:05.748882 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.748842    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1"} err="failed to get container status \"46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1\": rpc error: code = NotFound desc = could not find container \"46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1\": container with ID starting with 46ca719453ac810419b8ab7d3df0a7fb4bc1baa0ff454cbc8f194c950931ccb1 not found: ID does not exist"
Sep 09 11:06:05.760093 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.760067    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"]
Sep 09 11:06:05.766491 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:05.766465    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fe318cb6-kserve-74ccf4bdb7-w6x8q"]
Sep 09 11:06:06.475148 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:06.475113    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f020a1-4498-4171-8334-969f6d19fed8" path="/var/lib/kubelet/pods/10f020a1-4498-4171-8334-969f6d19fed8/volumes"
Sep 09 11:06:06.610907 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:06.610851    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:06:12.919322 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:12.919285    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:06:13.159571 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.159533    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-mhfz7"]
Sep 09 11:06:13.160056 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.160036    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f020a1-4498-4171-8334-969f6d19fed8" containerName="main"
Sep 09 11:06:13.160056 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.160057    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f020a1-4498-4171-8334-969f6d19fed8" containerName="main"
Sep 09 11:06:13.160218 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.160165    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="75bc31fa-0ca4-40a8-8c42-fe02414c2c3f" containerName="istio-proxy"
Sep 09 11:06:13.160218 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.160184    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f020a1-4498-4171-8334-969f6d19fed8" containerName="main"
Sep 09 11:06:13.164683 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.164662    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.167319 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.167291    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 11:06:13.168084 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.168059    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 11:06:13.168211 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.168067    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 11:06:13.168211 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.168067    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k8bm4\""
Sep 09 11:06:13.168211 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.168124    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 11:06:13.178678 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.178657    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-mhfz7"]
Sep 09 11:06:13.238689 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.238651    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-metrics-certs\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.238869 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.238697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-maas-api-tls\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.238869 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.238787    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbms\" (UniqueName: \"kubernetes.io/projected/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-kube-api-access-kwbms\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.238869 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.238815    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-openshift-service-ca\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.340110 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.340067    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbms\" (UniqueName: \"kubernetes.io/projected/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-kube-api-access-kwbms\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.340110 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.340110    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-openshift-service-ca\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.340392 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.340177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-metrics-certs\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.340392 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.340196    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-maas-api-tls\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.340815 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.340783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-openshift-service-ca\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.342761 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.342737    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-metrics-certs\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.342903 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.342865    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-maas-api-tls\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.356477 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.356445    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbms\" (UniqueName: \"kubernetes.io/projected/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-kube-api-access-kwbms\") pod \"maas-api-7998845878-mhfz7\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") " pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.475272 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.475184    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:13.753874 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.753846    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-mhfz7"]
Sep 09 11:06:13.756378 ip-10-0-131-250 kubenswrapper[2586]: W0909 11:06:13.756356    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c4065e_a2ef_4d65_a2b0_4b0b6b8891b9.slice/crio-5aeb9f6022d4b7a015e70ded45fe2b059898d6592417979b482ebd8f5ca99c04 WatchSource:0}: Error finding container 5aeb9f6022d4b7a015e70ded45fe2b059898d6592417979b482ebd8f5ca99c04: Status 404 returned error can't find the container with id 5aeb9f6022d4b7a015e70ded45fe2b059898d6592417979b482ebd8f5ca99c04
Sep 09 11:06:13.757774 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.757748    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 11:06:13.785090 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:13.785056    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7" event={"ID":"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9","Type":"ContainerStarted","Data":"5aeb9f6022d4b7a015e70ded45fe2b059898d6592417979b482ebd8f5ca99c04"}
Sep 09 11:06:14.793137 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:14.793042    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7" event={"ID":"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9","Type":"ContainerStarted","Data":"9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b"}
Sep 09 11:06:14.793519 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:14.793146    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:14.825950 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:14.825874    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7" podStartSLOduration=1.071496137 podStartE2EDuration="1.825860127s" totalImagesPullingTime="754.36399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:06:14.82514161 +0000 UTC m=+2416.880700660" watchObservedRunningTime="2026-09-09 11:06:14.825860127 +0000 UTC m=+2416.881419177"
Sep 09 11:06:20.803848 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:20.803772    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:20.836652 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:20.836615    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:06:28.420567 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:28.420529    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:06:34.006167 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.006132    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-mhfz7"]
Sep 09 11:06:34.006629 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.006367    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7" podUID="71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" containerName="maas-api" containerID="cri-o://9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b" gracePeriod=30
Sep 09 11:06:34.248995 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.248970    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:34.336770 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.336694    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-maas-api-tls\") pod \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") "
Sep 09 11:06:34.336770 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.336732    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-metrics-certs\") pod \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") "
Sep 09 11:06:34.336770 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.336757    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwbms\" (UniqueName: \"kubernetes.io/projected/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-kube-api-access-kwbms\") pod \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") "
Sep 09 11:06:34.337086 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.336786    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-openshift-service-ca\") pod \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\" (UID: \"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9\") "
Sep 09 11:06:34.337261 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.337228    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" (UID: "71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 11:06:34.338837 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.338816    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" (UID: "71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:06:34.338929 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.338851    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" (UID: "71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 11:06:34.338975 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.338932    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-kube-api-access-kwbms" (OuterVolumeSpecName: "kube-api-access-kwbms") pod "71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" (UID: "71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9"). InnerVolumeSpecName "kube-api-access-kwbms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 11:06:34.437490 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.437456    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:34.437490 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.437488    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:34.437490 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.437497    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwbms\" (UniqueName: \"kubernetes.io/projected/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-kube-api-access-kwbms\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:34.437490 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.437506    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 11:06:34.904941 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.904877    2586 generic.go:356] "Generic (PLEG): container finished" podID="71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" containerID="9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b" exitCode=0
Sep 09 11:06:34.904941 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.904933    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7" event={"ID":"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9","Type":"ContainerDied","Data":"9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b"}
Sep 09 11:06:34.905159 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.904971    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7"
Sep 09 11:06:34.905159 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.904993    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7998845878-mhfz7" event={"ID":"71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9","Type":"ContainerDied","Data":"5aeb9f6022d4b7a015e70ded45fe2b059898d6592417979b482ebd8f5ca99c04"}
Sep 09 11:06:34.905159 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.905016    2586 scope.go:117] "RemoveContainer" containerID="9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b"
Sep 09 11:06:34.914426 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.914408    2586 scope.go:117] "RemoveContainer" containerID="9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b"
Sep 09 11:06:34.914713 ip-10-0-131-250 kubenswrapper[2586]: E0909 11:06:34.914694    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b\": container with ID starting with 9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b not found: ID does not exist" containerID="9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b"
Sep 09 11:06:34.914769 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.914722    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b"} err="failed to get container status \"9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b\": rpc error: code = NotFound desc = could not find container \"9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b\": container with ID starting with 9d1f269dc6c4980a04dfda2b566cb127c21c82fe3dd7beacf835898ad02e697b not found: ID does not exist"
Sep 09 11:06:34.926681 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.926653    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-mhfz7"]
Sep 09 11:06:34.934370 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:34.934345    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7998845878-mhfz7"]
Sep 09 11:06:36.476403 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:36.476368    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" path="/var/lib/kubelet/pods/71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9/volumes"
Sep 09 11:06:36.711755 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:36.711718    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:06:44.120248 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:44.120216    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w5fxt"]
Sep 09 11:06:47.581287 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:47.581250    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-696d8d6478-k467s_ce43a5cb-a931-4444-a639-6a2ec9ce3257/authorino/0.log"
Sep 09 11:06:53.761085 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:53.761058    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xrhq2_c3c2bbd0-0d37-446b-a81d-023f3132c893/server/0.log"
Sep 09 11:06:53.896053 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:53.896022    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4rlxq_107da22e-8ba0-4b76-b5a1-b1b37acd952a/manager/0.log"
Sep 09 11:06:54.286625 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:54.286596    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-q4m59_eb1d453d-cab6-49d2-8a77-e7587479725a/manager/0.log"
Sep 09 11:06:54.303056 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:54.303029    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-q4m59_eb1d453d-cab6-49d2-8a77-e7587479725a/copy-manifests/0.log"
Sep 09 11:06:55.973190 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:55.973159    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-696d8d6478-k467s_ce43a5cb-a931-4444-a639-6a2ec9ce3257/authorino/0.log"
Sep 09 11:06:56.654133 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:56.654100    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w5fxt_17604753-84a4-413e-aac3-3dab4dca5ae7/limitador/0.log"
Sep 09 11:06:57.141752 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:57.141728    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb_bf219d22-e6ba-474a-879e-79b963df8772/istio-proxy/0.log"
Sep 09 11:06:57.365545 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:57.365518    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cd99c5fc-84px9_06a22e8b-0b98-44aa-87c7-0dc20095f080/kube-auth-proxy/0.log"
Sep 09 11:06:58.944983 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:58.944951    2586 scope.go:117] "RemoveContainer" containerID="e8bb3e5207aa7da16261da7f7cd5fd3babb8888567caf9296ce1a17ac70de6f9"
Sep 09 11:06:58.977360 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:06:58.977333    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-qsdh2_f6f32a40-dc73-461f-b2cb-52d88533eac0/main/0.log"
Sep 09 11:07:04.403305 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.403267    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-86rrh/must-gather-pdn7n"]
Sep 09 11:07:04.403697 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.403684    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" containerName="maas-api"
Sep 09 11:07:04.403697 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.403698    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" containerName="maas-api"
Sep 09 11:07:04.403771 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.403764    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c4065e-a2ef-4d65-a2b0-4b0b6b8891b9" containerName="maas-api"
Sep 09 11:07:04.415585 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.415555    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86rrh/must-gather-pdn7n"
Sep 09 11:07:04.418508 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.418486    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86rrh\"/\"kube-root-ca.crt\""
Sep 09 11:07:04.418825 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.418806    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86rrh\"/\"openshift-service-ca.crt\""
Sep 09 11:07:04.438236 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.438202    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86rrh/must-gather-pdn7n"]
Sep 09 11:07:04.518694 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.518656    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b74cc026-8c35-47d5-91c1-8a019a26ca2b-must-gather-output\") pod \"must-gather-pdn7n\" (UID: \"b74cc026-8c35-47d5-91c1-8a019a26ca2b\") " pod="openshift-must-gather-86rrh/must-gather-pdn7n"
Sep 09 11:07:04.518909 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.518804    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8s54\" (UniqueName: \"kubernetes.io/projected/b74cc026-8c35-47d5-91c1-8a019a26ca2b-kube-api-access-d8s54\") pod \"must-gather-pdn7n\" (UID: \"b74cc026-8c35-47d5-91c1-8a019a26ca2b\") " pod="openshift-must-gather-86rrh/must-gather-pdn7n"
Sep 09 11:07:04.620256 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.620213    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8s54\" (UniqueName: \"kubernetes.io/projected/b74cc026-8c35-47d5-91c1-8a019a26ca2b-kube-api-access-d8s54\") pod \"must-gather-pdn7n\" (UID: \"b74cc026-8c35-47d5-91c1-8a019a26ca2b\") " pod="openshift-must-gather-86rrh/must-gather-pdn7n"
Sep 09 11:07:04.620445 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.620304    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b74cc026-8c35-47d5-91c1-8a019a26ca2b-must-gather-output\") pod \"must-gather-pdn7n\" (UID: \"b74cc026-8c35-47d5-91c1-8a019a26ca2b\") " pod="openshift-must-gather-86rrh/must-gather-pdn7n"
Sep 09 11:07:04.620670 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.620648    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b74cc026-8c35-47d5-91c1-8a019a26ca2b-must-gather-output\") pod \"must-gather-pdn7n\" (UID: \"b74cc026-8c35-47d5-91c1-8a019a26ca2b\") " pod="openshift-must-gather-86rrh/must-gather-pdn7n"
Sep 09 11:07:04.643945 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.643920    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8s54\" (UniqueName: \"kubernetes.io/projected/b74cc026-8c35-47d5-91c1-8a019a26ca2b-kube-api-access-d8s54\") pod \"must-gather-pdn7n\" (UID: \"b74cc026-8c35-47d5-91c1-8a019a26ca2b\") " pod="openshift-must-gather-86rrh/must-gather-pdn7n"
Sep 09 11:07:04.724980 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:04.724876    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86rrh/must-gather-pdn7n"
Sep 09 11:07:05.024727 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:05.024705    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86rrh/must-gather-pdn7n"]
Sep 09 11:07:05.027156 ip-10-0-131-250 kubenswrapper[2586]: W0909 11:07:05.027121    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb74cc026_8c35_47d5_91c1_8a019a26ca2b.slice/crio-2d18655e6479c6a4a5357074a49887f498d3ebd8bf25edd1e4f9da73b306dcc6 WatchSource:0}: Error finding container 2d18655e6479c6a4a5357074a49887f498d3ebd8bf25edd1e4f9da73b306dcc6: Status 404 returned error can't find the container with id 2d18655e6479c6a4a5357074a49887f498d3ebd8bf25edd1e4f9da73b306dcc6
Sep 09 11:07:05.076918 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:05.076873    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/must-gather-pdn7n" event={"ID":"b74cc026-8c35-47d5-91c1-8a019a26ca2b","Type":"ContainerStarted","Data":"2d18655e6479c6a4a5357074a49887f498d3ebd8bf25edd1e4f9da73b306dcc6"}
Sep 09 11:07:07.092111 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:07.092072    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/must-gather-pdn7n" event={"ID":"b74cc026-8c35-47d5-91c1-8a019a26ca2b","Type":"ContainerStarted","Data":"796b53b5cf7272fe0065a10b1c284cdecb09296a5a26d5cc537da04bf2a269f8"}
Sep 09 11:07:07.092584 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:07.092119    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/must-gather-pdn7n" event={"ID":"b74cc026-8c35-47d5-91c1-8a019a26ca2b","Type":"ContainerStarted","Data":"0ab3cd10f3699d07061a356b1cf45e77d9fd2faddd8dbf64f6e5dce85f7539e8"}
Sep 09 11:07:07.961511 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:07.961476    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4bqkk_2028e44e-02bc-44ab-805f-135d6b34343e/global-pull-secret-syncer/0.log"
Sep 09 11:07:08.209627 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:08.209573    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mgvw6_38cd53b9-6167-4f5b-a272-baa8f3a674f3/konnectivity-agent/0.log"
Sep 09 11:07:08.270415 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:08.270335    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-250.ec2.internal_b918f93a5380fe0ffc07a388ced93b00/haproxy/0.log"
Sep 09 11:07:12.797133 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:12.797090    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-696d8d6478-k467s_ce43a5cb-a931-4444-a639-6a2ec9ce3257/authorino/0.log"
Sep 09 11:07:13.013790 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:13.013755    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w5fxt_17604753-84a4-413e-aac3-3dab4dca5ae7/limitador/0.log"
Sep 09 11:07:14.710752 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:14.710703    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678bf4b7-00cc-4db5-afa4-1eefb60c8016/alertmanager/0.log"
Sep 09 11:07:14.773035 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:14.772999    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678bf4b7-00cc-4db5-afa4-1eefb60c8016/config-reloader/0.log"
Sep 09 11:07:14.819760 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:14.819690    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678bf4b7-00cc-4db5-afa4-1eefb60c8016/kube-rbac-proxy-web/0.log"
Sep 09 11:07:14.895467 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:14.895420    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678bf4b7-00cc-4db5-afa4-1eefb60c8016/kube-rbac-proxy/0.log"
Sep 09 11:07:15.005438 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:15.005356    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678bf4b7-00cc-4db5-afa4-1eefb60c8016/kube-rbac-proxy-metric/0.log"
Sep 09 11:07:15.230391 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:15.230186    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678bf4b7-00cc-4db5-afa4-1eefb60c8016/prom-label-proxy/0.log"
Sep 09 11:07:15.322580 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:15.322496    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678bf4b7-00cc-4db5-afa4-1eefb60c8016/init-config-reloader/0.log"
Sep 09 11:07:15.422144 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:15.422099    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-lm2ph_fd142694-acce-48c2-ba76-c2361de63250/cluster-monitoring-operator/0.log"
Sep 09 11:07:15.634508 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:15.634478    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-588d844b79-6xj8d_9848bce8-6ce5-47a3-839d-ebd153b2ed07/metrics-server/0.log"
Sep 09 11:07:15.704087 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:15.704047    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-vhwp8_d440b101-26a7-49e8-a737-c1b65eb56093/monitoring-plugin/0.log"
Sep 09 11:07:15.936314 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:15.936213    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjrpb_21911b8f-8acf-446d-a04f-b5f367928408/node-exporter/0.log"
Sep 09 11:07:16.003050 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:16.002995    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjrpb_21911b8f-8acf-446d-a04f-b5f367928408/kube-rbac-proxy/0.log"
Sep 09 11:07:16.039437 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:16.039408    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjrpb_21911b8f-8acf-446d-a04f-b5f367928408/init-textfile/0.log"
Sep 09 11:07:16.948599 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:16.948564    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7cc9b8d685-jjf8p_077d1c94-d212-467f-a555-182fcd143100/telemeter-client/0.log"
Sep 09 11:07:17.001182 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.001150    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7cc9b8d685-jjf8p_077d1c94-d212-467f-a555-182fcd143100/reload/0.log"
Sep 09 11:07:17.093460 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.093433    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7cc9b8d685-jjf8p_077d1c94-d212-467f-a555-182fcd143100/kube-rbac-proxy/0.log"
Sep 09 11:07:17.172706 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.172677    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b975d59c4-qp5jn_c688af9f-4254-46ce-89c1-732ca4651855/thanos-query/0.log"
Sep 09 11:07:17.211052 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.210968    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b975d59c4-qp5jn_c688af9f-4254-46ce-89c1-732ca4651855/kube-rbac-proxy-web/0.log"
Sep 09 11:07:17.256640 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.256603    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b975d59c4-qp5jn_c688af9f-4254-46ce-89c1-732ca4651855/kube-rbac-proxy/0.log"
Sep 09 11:07:17.299799 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.299771    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b975d59c4-qp5jn_c688af9f-4254-46ce-89c1-732ca4651855/prom-label-proxy/0.log"
Sep 09 11:07:17.337198 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.337143    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b975d59c4-qp5jn_c688af9f-4254-46ce-89c1-732ca4651855/kube-rbac-proxy-rules/0.log"
Sep 09 11:07:17.364821 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.364791    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b975d59c4-qp5jn_c688af9f-4254-46ce-89c1-732ca4651855/kube-rbac-proxy-metrics/0.log"
Sep 09 11:07:17.451713 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.451639    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-86rrh/must-gather-pdn7n" podStartSLOduration=12.479082446 podStartE2EDuration="13.451614458s" totalImagesPullingTime="972.532012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:07:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:07:07.126521758 +0000 UTC m=+2469.182080809" watchObservedRunningTime="2026-09-09 11:07:17.451614458 +0000 UTC m=+2479.507173508"
Sep 09 11:07:17.451976 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.451950    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"]
Sep 09 11:07:17.458145 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.458111    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.461207 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.461100    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-86rrh\"/\"default-dockercfg-l6fff\""
Sep 09 11:07:17.473033 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.472998    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"]
Sep 09 11:07:17.548856 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.548823    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-sys\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.548856 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.548861    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-proc\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.549135 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.549041    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjk7p\" (UniqueName: \"kubernetes.io/projected/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-kube-api-access-kjk7p\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.549135 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.549092    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-lib-modules\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.549259 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.549223    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-podres\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.650524 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650486    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-lib-modules\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.650723 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650556    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-podres\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.650723 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650594    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-sys\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.650723 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650620    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-proc\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.650723 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650689    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjk7p\" (UniqueName: \"kubernetes.io/projected/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-kube-api-access-kjk7p\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.650723 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650706    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-lib-modules\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.651027 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650724    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-podres\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.651027 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650705    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-sys\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.651027 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.650753    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-proc\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.659486 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.659456    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjk7p\" (UniqueName: \"kubernetes.io/projected/b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9-kube-api-access-kjk7p\") pod \"perf-node-gather-daemonset-d5h8f\" (UID: \"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:17.775778 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:17.775681    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:18.083937 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:18.082704    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"]
Sep 09 11:07:18.086589 ip-10-0-131-250 kubenswrapper[2586]: W0909 11:07:18.086512    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb06ac377_d76b_4bc7_98d9_2ae0ad18ebe9.slice/crio-fab58e6e3f9737847899183c45cfc9304576f914d7d756267ca4f7d7e1506fbb WatchSource:0}: Error finding container fab58e6e3f9737847899183c45cfc9304576f914d7d756267ca4f7d7e1506fbb: Status 404 returned error can't find the container with id fab58e6e3f9737847899183c45cfc9304576f914d7d756267ca4f7d7e1506fbb
Sep 09 11:07:18.214247 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:18.214207    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f" event={"ID":"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9","Type":"ContainerStarted","Data":"fab58e6e3f9737847899183c45cfc9304576f914d7d756267ca4f7d7e1506fbb"}
Sep 09 11:07:18.551498 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:18.551452    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-jpm9t_4f9527ed-c3ba-45eb-a51c-80922f29870a/networking-console-plugin/0.log"
Sep 09 11:07:19.225968 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:19.225927    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f" event={"ID":"b06ac377-d76b-4bc7-98d9-2ae0ad18ebe9","Type":"ContainerStarted","Data":"63a2fbab4d8a4322a8aba069204ab50f0a4285240d44e88f8b32d84a0b330771"}
Sep 09 11:07:19.226368 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:19.226123    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:19.246406 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:19.246345    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f" podStartSLOduration=2.246321124 podStartE2EDuration="2.246321124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 11:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:07:19.243169047 +0000 UTC m=+2481.298728098" watchObservedRunningTime="2026-09-09 11:07:19.246321124 +0000 UTC m=+2481.301880175"
Sep 09 11:07:19.761128 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:19.761096    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7755c8cfcb-7ghqf_5721e109-5915-4438-b897-4ac7a9c5458e/console/0.log"
Sep 09 11:07:20.463006 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:20.462960    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-cwb4w_018e1efe-ab8e-4a08-a366-56aab8b91458/volume-data-source-validator/0.log"
Sep 09 11:07:21.675338 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:21.675309    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4c2kf_4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226/dns/0.log"
Sep 09 11:07:21.706246 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:21.706221    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4c2kf_4b2d6d3d-fb6c-4aa1-b70e-839cf96a3226/kube-rbac-proxy/0.log"
Sep 09 11:07:21.949414 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:21.949336    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flzz6_4737cd63-ad67-44bf-9164-f1a50aa8eda7/dns-node-resolver/0.log"
Sep 09 11:07:22.676743 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:22.676691    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dl5gz_c4acaab0-9fe1-4c34-8bb9-45ba5d8bf886/node-ca/0.log"
Sep 09 11:07:23.820591 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:23.820551    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb_bf219d22-e6ba-474a-879e-79b963df8772/istio-proxy/0.log"
Sep 09 11:07:24.085221 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:24.085150    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cd99c5fc-84px9_06a22e8b-0b98-44aa-87c7-0dc20095f080/kube-auth-proxy/0.log"
Sep 09 11:07:25.246676 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:25.245711    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-d5h8f"
Sep 09 11:07:25.582005 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:25.581917    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zznrb_ad647ae4-c0ad-4e78-a358-cbfb757329f0/serve-healthcheck-canary/0.log"
Sep 09 11:07:26.224658 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:26.224619    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-rmszz_db6c7c40-96e8-473e-ae77-35168a674dbe/insights-operator/0.log"
Sep 09 11:07:26.225394 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:26.225366    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-rmszz_db6c7c40-96e8-473e-ae77-35168a674dbe/insights-operator/1.log"
Sep 09 11:07:26.255308 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:26.255271    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lpgj4_68170ad4-92bb-4332-aab6-831afb919a87/kube-rbac-proxy/0.log"
Sep 09 11:07:26.284108 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:26.284086    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lpgj4_68170ad4-92bb-4332-aab6-831afb919a87/exporter/0.log"
Sep 09 11:07:26.317714 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:26.317684    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lpgj4_68170ad4-92bb-4332-aab6-831afb919a87/extractor/0.log"
Sep 09 11:07:28.271065 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:28.271033    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb_bf219d22-e6ba-474a-879e-79b963df8772/istio-proxy/0.log"
Sep 09 11:07:28.509985 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:28.509956    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cd99c5fc-84px9_06a22e8b-0b98-44aa-87c7-0dc20095f080/kube-auth-proxy/0.log"
Sep 09 11:07:29.461778 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:29.461746    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xrhq2_c3c2bbd0-0d37-446b-a81d-023f3132c893/server/0.log"
Sep 09 11:07:29.579835 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:29.579792    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4rlxq_107da22e-8ba0-4b76-b5a1-b1b37acd952a/manager/0.log"
Sep 09 11:07:29.895701 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:29.895667    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-q4m59_eb1d453d-cab6-49d2-8a77-e7587479725a/manager/0.log"
Sep 09 11:07:29.942230 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:29.942202    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-q4m59_eb1d453d-cab6-49d2-8a77-e7587479725a/copy-manifests/0.log"
Sep 09 11:07:30.148741 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:30.148693    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6d7hb_bf219d22-e6ba-474a-879e-79b963df8772/istio-proxy/0.log"
Sep 09 11:07:30.396725 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:30.396682    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cd99c5fc-84px9_06a22e8b-0b98-44aa-87c7-0dc20095f080/kube-auth-proxy/0.log"
Sep 09 11:07:31.275870 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:31.275838    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7658745f98-pgwlc_7cd10489-77ac-48fe-bed6-ab99f84a2c18/manager/0.log"
Sep 09 11:07:31.369767 ip-10-0-131-250 kubenswrapper[2586]: I0909 11:07:31.369734    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gxvwn_72d8059e-141a-46a3-ac60-7f7b45fcf36d/openshift-lws-operator/0.log"