Sep 15 21:07:59.973806 ip-10-0-140-113 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 21:08:00.423932 ip-10-0-140-113 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:08:00.423932 ip-10-0-140-113 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 21:08:00.423932 ip-10-0-140-113 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:08:00.423932 ip-10-0-140-113 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 21:08:00.423932 ip-10-0-140-113 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:08:00.426014 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.425914    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 21:08:00.430313 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430291    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:00.430313 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430310    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:00.430313 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430315    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:00.430313 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430320    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430324    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430328    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430332    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430335    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430339    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430343    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430347    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430350    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430354    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430358    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430362    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430366    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430371    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430374    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430388    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430393    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430397    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430401    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430405    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:00.430592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430408    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430413    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430417    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430421    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430424    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430428    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430432    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430436    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430440    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430444    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430448    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430453    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430457    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430478    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430487    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430492    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430496    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430500    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430504    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:00.431454 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430511    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430518    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430526    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430531    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430535    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430539    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430544    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430548    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430552    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430557    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430562    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430566    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430570    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430574    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430579    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430582    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430587    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430591    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430595    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:00.432274 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430598    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430602    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430607    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430611    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430617    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430621    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430625    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430629    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430635    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430640    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430644    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430648    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430653    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430656    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430660    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430664    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430669    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430673    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430677    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430681    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:00.432843 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430685    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430689    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430693    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430698    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430702    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430706    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430710    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430714    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430718    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430722    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430726    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430731    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430735    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430739    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430746    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430751    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430756    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.430761    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431949    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:00.433364 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431959    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431965    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431969    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431973    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431978    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431982    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431986    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.431993    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432000    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432005    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432009    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432013    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432018    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432022    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432026    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432031    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432035    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432040    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432044    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432048    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:00.433861 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432053    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432057    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432061    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432065    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432069    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432074    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432078    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432083    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432087    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432092    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432096    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432100    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432104    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432108    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432113    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432117    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432121    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432125    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432130    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:00.434360 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432134    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432138    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432142    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432146    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432152    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432156    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432160    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432164    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432169    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432173    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432177    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432182    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432187    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432191    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432195    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432199    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432205    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432209    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432213    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432217    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:00.434851 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432222    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432227    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432231    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432235    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432239    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432243    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432248    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432252    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432256    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432262    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432266    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432270    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432274    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432278    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432281    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432285    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432289    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432294    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432297    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432301    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:00.435342 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432305    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432309    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432314    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432318    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432323    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432330    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432336    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432341    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432345    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432349    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432354    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432358    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432363    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432367    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432371    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432375    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432381    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432385    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432390    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:00.435872 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432909    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432922    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432925    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432929    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432932    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432935    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432938    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432941    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432943    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432946    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432949    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432952    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432955    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432958    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432960    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432963    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432965    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432968    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432974    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432976    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:00.436344 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432979    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432982    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432984    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432987    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432991    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432995    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.432999    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433001    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433004    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433007    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433009    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433012    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433015    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433018    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433021    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433024    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433026    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433029    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433031    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:00.436850 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433041    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433043    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433046    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433050    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433054    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433057    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433060    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433063    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433066    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433068    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433071    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433075    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433077    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433081    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433083    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433086    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433088    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433091    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433094    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433096    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:00.437352 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433099    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433101    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433104    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433106    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433109    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433114    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433117    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433120    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433122    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433125    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433127    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433132    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433136    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433138    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433141    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433144    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433146    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433149    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433152    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433154    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:00.437863 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433157    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433159    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433162    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433166    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433168    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433171    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433173    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433176    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433178    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433181    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433183    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433186    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433189    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433191    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433194    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433196    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433199    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433203    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433205    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.433208    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:00.438338 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433294    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433332    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433340    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433344    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433348    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433352    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433356    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433360    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433364    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433367    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433371    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433376    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433379    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433382    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433385    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433388    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433392    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433395    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433399    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433402    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433405    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433408    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433411    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433415    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 21:08:00.438887 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433418    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433421    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433424    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433428    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433431    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433435    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433438    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433441    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433445    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433448    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433451    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433454    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433457    2563 flags.go:64] FLAG: --enable-server="true"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433460    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433480    2563 flags.go:64] FLAG: --event-burst="100"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433483    2563 flags.go:64] FLAG: --event-qps="50"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433486    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433489    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433492    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433496    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433499    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433502    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433505    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433508    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433511    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 21:08:00.439478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433514    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433517    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433520    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433523    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433526    2563 flags.go:64] FLAG: --feature-gates=""
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433530    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433532    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433535    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433538    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433541    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433544    2563 flags.go:64] FLAG: --help="false"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433547    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433552    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433555    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433558    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433561    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433565    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433568    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433570    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433573    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433576    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433579    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433582    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433585    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 15 21:08:00.440108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433588    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433591    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433594    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433597    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433600    2563 flags.go:64] FLAG: --lock-file=""
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433603    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433606    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433608    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433614    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433616    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433619    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433622    2563 flags.go:64] FLAG: --logging-format="text"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433625    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433628    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433631    2563 flags.go:64] FLAG: --manifest-url=""
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433634    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433638    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433641    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433645    2563 flags.go:64] FLAG: --max-pods="110"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433648    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433651    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433654    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433657    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433660    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433663    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 21:08:00.440713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433666    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433674    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433677    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433680    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433683    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433686    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433692    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433695    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433698    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433700    2563 flags.go:64] FLAG: --port="10250"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433703    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433706    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c8008594bfc01e83"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433710    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433713    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433715    2563 flags.go:64] FLAG: --register-node="true"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433718    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433721    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433724    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433727    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433730    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433733    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433736    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433739    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433742    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433744    2563 flags.go:64] FLAG: --runonce="false"
Sep 15 21:08:00.441350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433747    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433750    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433753    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433757    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433760    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433763    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433766    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433769    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433772    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433775    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433778    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433780    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433783    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433786    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433789    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433794    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433797    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433800    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433803    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433807    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433811    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433814    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433817    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433820    2563 flags.go:64] FLAG: --v="2"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433824    2563 flags.go:64] FLAG: --version="false"
Sep 15 21:08:00.442092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433828    2563 flags.go:64] FLAG: --vmodule=""
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433831    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.433834    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434011    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434016    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434019    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434021    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434024    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434027    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434030    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434033    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434036    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434039    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434042    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434044    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434047    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434049    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434052    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434054    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434057    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:00.443106 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434060    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434062    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434065    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434067    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434070    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434072    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434075    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434081    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434084    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434086    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434089    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434091    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434094    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434096    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434099    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434101    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434104    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434106    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434108    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:00.443652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434111    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434113    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434116    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434119    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434122    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434125    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434129    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434132    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434134    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434138    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434142    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434145    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434148    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434151    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434153    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434156    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434158    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434161    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434163    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:00.444113 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434165    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434169    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434172    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434175    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434177    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434180    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434183    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434185    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434187    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434190    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434193    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434195    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434197    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434200    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434202    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434205    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434208    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434211    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434213    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434216    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434219    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:00.444690 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434222    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434224    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434227    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434229    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434232    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434235    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434238    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434240    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434243    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434245    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434248    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434251    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434256    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434260    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434266    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434270    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434274    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434277    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434280    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434283    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:00.445218 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434286    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434289    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.434291    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.434296    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.444170    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.444192    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.444207    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.444214    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444277    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444283    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444286    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444289    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444292    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444294    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:00.445815 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444297    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444300    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444303    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444305    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444307    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444310    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444313    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444315    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444318    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444321    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444323    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444326    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444328    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444331    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444333    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444336    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444338    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444341    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444344    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444346    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:00.446211 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444350    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444354    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444362    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444366    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444369    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444371    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444375    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444378    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444380    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444383    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444385    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444389    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444393    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444396    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444399    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444401    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444404    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444411    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444413    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444416    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:00.446713 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444419    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444422    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444425    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444427    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444430    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444432    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444435    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444437    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444440    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444442    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444445    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444447    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444450    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444452    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444455    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444459    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444481    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444487    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444491    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444494    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:00.447197 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444497    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444499    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444502    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444505    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444508    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444511    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444515    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444518    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444521    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444524    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444527    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444530    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444533    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444538    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444541    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444544    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444546    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444549    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444551    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:00.447707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444554    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444556    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444558    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444561    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444564    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444566    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444569    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444571    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444574    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444577    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444579    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444582    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444584    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444587    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.444592    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:08:00.448164 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444717    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444722    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444725    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444728    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444731    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444734    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444736    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444739    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444743    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444747    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444750    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444753    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444756    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444759    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444761    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444764    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444767    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444770    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444772    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444774    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:00.448543 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444777    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444779    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444782    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444784    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444787    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444789    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444792    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444795    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444797    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444800    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444802    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444805    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444808    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444810    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444813    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444815    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444818    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444820    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444824    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444827    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:00.449016 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444830    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444832    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444835    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444837    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444840    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444843    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444846    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444848    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444851    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444853    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444855    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444858    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444861    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444863    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444866    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444868    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444871    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444873    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444876    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444878    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:00.449650 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444881    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444883    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444886    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444889    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444892    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444894    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444896    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444899    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444902    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444906    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444909    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444911    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444914    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444917    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444920    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444923    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444926    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444928    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444932    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:00.450140 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444935    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444937    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444940    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444943    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444945    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444948    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444951    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444954    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444957    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444959    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444961    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444964    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444967    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444969    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444971    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444974    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444976    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444981    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444984    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:00.450618 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:00.444987    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:00.451079 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.444992    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:08:00.451079 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.445133    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 21:08:00.451296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.451279    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 21:08:00.452345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.452333    2563 server.go:1019] "Starting client certificate rotation"
Sep 15 21:08:00.452456 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.452436    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:08:00.452506 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.452499    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:08:00.455360 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.455346    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:08:00.456506 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.456489    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:08:00.469963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.469937    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 15 21:08:00.474508 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.474491    2563 log.go:25] "Validated CRI v1 image API"
Sep 15 21:08:00.475497 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.475478    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 21:08:00.479186 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.479164    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 92ff4c76-a814-4f25-a768-03e9c0e1ece6:/dev/nvme0n1p3 9c1c297d-1b2f-4047-a076-085097b82ac1:/dev/nvme0n1p4]
Sep 15 21:08:00.479252 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.479184    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 21:08:00.486340 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.486207    2563 manager.go:216] Machine: {Timestamp:2026-09-15 21:08:00.483643117 +0000 UTC m=+0.380536017 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115466 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29f550b0d75edaec8d162005f95ffb SystemUUID:ec29f550-b0d7-5eda-ec8d-162005f95ffb BootID:e6940ce9-05da-4cae-a13f-d8d217b3504b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:82:47:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:82:47:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:5a:c3:34:07:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 21:08:00.486340 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.486330    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 21:08:00.486517 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.486503    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 21:08:00.491242 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.491215    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:08:00.491705 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.491393    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 21:08:00.491877 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.491716    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-113.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 21:08:00.493349 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.493336    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 21:08:00.493414 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.493357    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 21:08:00.493414 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.493369    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:08:00.494198 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.494186    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:08:00.495411 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.495398    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 21:08:00.500075 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500060    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:08:00.500219 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500208    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 21:08:00.500279 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500248    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 21:08:00.500279 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500263    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 21:08:00.500363 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500286    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 21:08:00.500363 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500301    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 15 21:08:00.500363 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500313    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 21:08:00.500723 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500710    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:08:00.500795 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.500734    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:08:00.503099 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.503082    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 21:08:00.505942 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.505927    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 21:08:00.505991 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.505959    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 21:08:00.508010 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.507997    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 21:08:00.508065 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508016    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 21:08:00.508065 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508023    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 21:08:00.508065 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508029    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 21:08:00.508065 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508037    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 21:08:00.508065 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508043    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 21:08:00.508201 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508073    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 21:08:00.508201 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508080    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 21:08:00.508201 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508087    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 21:08:00.508201 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508093    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 21:08:00.508201 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508102    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 21:08:00.508201 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.508108    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 21:08:00.509015 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.509005    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 21:08:00.509015 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.509016    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 21:08:00.513020 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.512998    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-113.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 21:08:00.513163 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.513150    2563 server.go:1268] "Started kubelet"
Sep 15 21:08:00.513308 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.513282    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 21:08:00.513365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.513276    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 21:08:00.513365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.513348    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 21:08:00.513859 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.513822    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-113.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 21:08:00.513859 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.513824    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 21:08:00.514013 ip-10-0-140-113 systemd[1]: Started Kubernetes Kubelet.
Sep 15 21:08:00.514942 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.514821    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 21:08:00.514942 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.514857    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 21:08:00.518479 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.518446    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 21:08:00.520052 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.518810    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-113.ec2.internal.18d59aa9ae3e8043  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-113.ec2.internal,UID:ip-10-0-140-113.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-113.ec2.internal,},FirstTimestamp:2026-09-15 21:08:00.513122371 +0000 UTC m=+0.410015272,LastTimestamp:2026-09-15 21:08:00.513122371 +0000 UTC m=+0.410015272,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-113.ec2.internal,}"
Sep 15 21:08:00.522321 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.522304    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 21:08:00.523058 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.523043    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 21:08:00.523243 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.523205    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:00.524010 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.523993    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 21:08:00.524107 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.523994    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 21:08:00.524107 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.524034    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 21:08:00.524211 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.524171    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 21:08:00.524211 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.524180    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 21:08:00.524323 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.524304    2563 factory.go:55] Registering systemd factory
Sep 15 21:08:00.524355 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.524328    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 15 21:08:00.525456 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.525441    2563 factory.go:153] Registering CRI-O factory
Sep 15 21:08:00.525456 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.525456    2563 factory.go:223] Registration of the crio container factory successfully
Sep 15 21:08:00.525575 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.525561    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 21:08:00.525626 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.525587    2563 factory.go:103] Registering Raw factory
Sep 15 21:08:00.525626 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.525598    2563 manager.go:1196] Started watching for new ooms in manager
Sep 15 21:08:00.526135 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.526121    2563 manager.go:349] Starting recovery of all containers
Sep 15 21:08:00.530254 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.530224    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-113.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 21:08:00.530254 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.530227    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 21:08:00.539850 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.539832    2563 manager.go:354] Recovery completed
Sep 15 21:08:00.544042 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.544029    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:00.546102 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.546082    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bn2px"
Sep 15 21:08:00.546277 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.546260    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:00.546348 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.546288    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:00.546348 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.546298    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:00.546913 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.546900    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 21:08:00.546913 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.546905    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:00.546913 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.546911    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 21:08:00.547055 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.546926    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:08:00.548934 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.548867    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-113.ec2.internal.18d59aa9b0386321  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-113.ec2.internal,UID:ip-10-0-140-113.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-113.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-113.ec2.internal,},FirstTimestamp:2026-09-15 21:08:00.546276129 +0000 UTC m=+0.443169029,LastTimestamp:2026-09-15 21:08:00.546276129 +0000 UTC m=+0.443169029,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-113.ec2.internal,}"
Sep 15 21:08:00.549193 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.549177    2563 policy_none.go:49] "None policy: Start"
Sep 15 21:08:00.549289 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.549196    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 21:08:00.549289 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.549210    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 21:08:00.550026 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.550009    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:00.550080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.550034    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:00.550080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.550044    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:00.551265 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.551254    2563 policy_none.go:47] "Start"
Sep 15 21:08:00.555432 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.555415    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bn2px"
Sep 15 21:08:00.561163 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.561074    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-113.ec2.internal.18d59aa9b038a456  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-113.ec2.internal,UID:ip-10-0-140-113.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-113.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-113.ec2.internal,},FirstTimestamp:2026-09-15 21:08:00.546292822 +0000 UTC m=+0.443185721,LastTimestamp:2026-09-15 21:08:00.546292822 +0000 UTC m=+0.443185721,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-113.ec2.internal,}"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.595243    2563 manager.go:337] "Starting Device Plugin manager"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.595277    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.595286    2563 server.go:87] "Starting device plugin registration server"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.595446    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.595688    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.595713    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.595901    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.595995    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.596004    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.597055    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.597106    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.597131    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.597175    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.597391    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.597446    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:00.605569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.598762    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:00.696930 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.696818    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:00.697304 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.697271    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal"]
Sep 15 21:08:00.697391 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.697360    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:00.697957 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.697937    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:00.698029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.697970    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:00.698029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.697984    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:00.698029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.698011    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.698822 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.698806    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:00.698911 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.698836    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:00.698911 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.698850    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:00.701248 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.701233    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:00.701381 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.701358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.701423 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.701389    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:00.702039 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.702022    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:00.702105 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.702054    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:00.702105 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.702066    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:00.702169 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.702022    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:00.702169 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.702134    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:00.702169 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.702150    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:00.704387 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.704363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.704447 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.704399    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:00.705207 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.705193    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:00.705285 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.705220    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:00.705285 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.705233    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:00.706732 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.706718    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.706775 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.706743    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-113.ec2.internal\": node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:00.725688 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.725662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7cf5090aadfc7dceb51f9b0a797cc1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal\" (UID: \"da7cf5090aadfc7dceb51f9b0a797cc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.725795 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.725697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab1af32928d1c75375f28ea688cf59c1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-113.ec2.internal\" (UID: \"ab1af32928d1c75375f28ea688cf59c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.725795 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.725714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da7cf5090aadfc7dceb51f9b0a797cc1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal\" (UID: \"da7cf5090aadfc7dceb51f9b0a797cc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.731355 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.731338    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-113.ec2.internal\" not found" node="ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.735914 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.735899    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-113.ec2.internal\" not found" node="ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.750674 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.750652    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:00.826460 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.826432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da7cf5090aadfc7dceb51f9b0a797cc1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal\" (UID: \"da7cf5090aadfc7dceb51f9b0a797cc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.826460 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.826461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7cf5090aadfc7dceb51f9b0a797cc1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal\" (UID: \"da7cf5090aadfc7dceb51f9b0a797cc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.826657 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.826500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab1af32928d1c75375f28ea688cf59c1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-113.ec2.internal\" (UID: \"ab1af32928d1c75375f28ea688cf59c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.826657 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.826546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab1af32928d1c75375f28ea688cf59c1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-113.ec2.internal\" (UID: \"ab1af32928d1c75375f28ea688cf59c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.826657 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.826553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7cf5090aadfc7dceb51f9b0a797cc1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal\" (UID: \"da7cf5090aadfc7dceb51f9b0a797cc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.826657 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:00.826551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da7cf5090aadfc7dceb51f9b0a797cc1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal\" (UID: \"da7cf5090aadfc7dceb51f9b0a797cc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:00.850887 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.850862    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:00.951460 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:00.951385    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.035239 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.035204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:01.039432 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.039412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:01.051730 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.051710    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.152313 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.152272    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.252904 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.252838    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.353543 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.353504    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.452128 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.452091    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 21:08:01.452600 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.452239    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:08:01.454252 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.454230    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.554609 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.554523    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.559704 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.559662    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:03:01 +0000 UTC" deadline="2028-03-26 12:10:42.415113128 +0000 UTC"
Sep 15 21:08:01.559764 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.559706    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13383h2m40.855410841s"
Sep 15 21:08:01.582880 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.582852    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:01.583929 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.583913    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:01.583996 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.583945    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:01.583996 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.583957    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:01.655409 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.655318    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.755890 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.755854    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.763840 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.763676    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:01.772798 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:01.772763    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7cf5090aadfc7dceb51f9b0a797cc1.slice/crio-68110e05b4619b86f678934eff5c15116f75af3edd0eed79dccb5769390b3ed9 WatchSource:0}: Error finding container 68110e05b4619b86f678934eff5c15116f75af3edd0eed79dccb5769390b3ed9: Status 404 returned error can't find the container with id 68110e05b4619b86f678934eff5c15116f75af3edd0eed79dccb5769390b3ed9
Sep 15 21:08:01.777115 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.777101    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:08:01.856364 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:01.856279    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-113.ec2.internal\" not found"
Sep 15 21:08:01.923583 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:01.923554    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:03.316671 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.316642    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:03.320394 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.320376    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:03.327836 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.327821    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:08:03.328849 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.328835    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal"
Sep 15 21:08:03.343711 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.343688    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:08:03.425761 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:03.425729    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1af32928d1c75375f28ea688cf59c1.slice/crio-b74003f2737a1287ef782328cd0992e86a2aa638da72b6c11bb39e7cfa9b9b7d WatchSource:0}: Error finding container b74003f2737a1287ef782328cd0992e86a2aa638da72b6c11bb39e7cfa9b9b7d: Status 404 returned error can't find the container with id b74003f2737a1287ef782328cd0992e86a2aa638da72b6c11bb39e7cfa9b9b7d
Sep 15 21:08:03.586162 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.586132    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:03.798897 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.798801    2563 apiserver.go:52] "Watching apiserver"
Sep 15 21:08:03.805635 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.805583    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 21:08:03.806151 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.806126    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-xk55v","openshift-network-operator/iptables-alerter-wbdhj","kube-system/global-pull-secret-syncer-tcvp4","kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49","openshift-image-registry/node-ca-lmfnh","openshift-multus/network-metrics-daemon-jlctj","openshift-network-diagnostics/network-check-target-w5n4g","openshift-ovn-kubernetes/ovnkube-node-fxcmb","kube-system/konnectivity-agent-xthx4","openshift-cluster-node-tuning-operator/tuned-4jmnc","openshift-dns/node-resolver-zngks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal","openshift-multus/multus-additional-cni-plugins-7pdpq"]
Sep 15 21:08:03.808624 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.808602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:03.810725 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.810702    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 21:08:03.810820 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.810804    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5hd2v\""
Sep 15 21:08:03.810900 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.810707    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 21:08:03.811197 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.811178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.812869 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.812847    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:08:03.812962 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.812894    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 21:08:03.813036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.813019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vksqf\""
Sep 15 21:08:03.813723 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.813597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.813723 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:03.813673    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:03.816431 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.816413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.819609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.819878    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.819949    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.820234    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.820242    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.820515    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.820572    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2tzc4\""
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.821209    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.821552    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.821992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.823377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.822323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 21:08:03.823981 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.823788    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 21:08:03.823981 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.823868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wm2rn\""
Sep 15 21:08:03.824078 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.824010    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 21:08:03.824128 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.824100    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 21:08:03.824179 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.824131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 21:08:03.824268 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.824252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hv2xw\""
Sep 15 21:08:03.825472 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.825283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.827765 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.827739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:03.827849 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.827822    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zwml2\""
Sep 15 21:08:03.827968 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.827948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 21:08:03.828049 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:03.827816    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:03.828136 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.828116    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 21:08:03.828731 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.828363    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 21:08:03.828731 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.828407    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 21:08:03.828731 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.828618    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 21:08:03.830738 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.830072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:03.831126 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.831105    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 21:08:03.831902 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.831794    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:08:03.832403 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.832382    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7259v\""
Sep 15 21:08:03.832579 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.832561    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 21:08:03.832680 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.832661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.833277 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833248    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 21:08:03.833510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3e119ff-4a7f-4b5e-aafe-c48b610649af-konnectivity-ca\") pod \"konnectivity-agent-xthx4\" (UID: \"a3e119ff-4a7f-4b5e-aafe-c48b610649af\") " pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:03.833593 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833523    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:08:03.833593 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.833593 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833572    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.833790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-sys\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.833790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-lib-modules\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.833790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9dd\" (UniqueName: \"kubernetes.io/projected/b7549f02-e0b7-4791-acf5-cee526852cca-kube-api-access-4l9dd\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.833790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3e119ff-4a7f-4b5e-aafe-c48b610649af-agent-certs\") pod \"konnectivity-agent-xthx4\" (UID: \"a3e119ff-4a7f-4b5e-aafe-c48b610649af\") " pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:03.833790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysctl-d\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.833790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-var-lib-kubelet\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7549f02-e0b7-4791-acf5-cee526852cca-etc-tuned\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.834074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-sys-fs\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.834074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-serviceca\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.834074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-modprobe-d\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.833994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysconfig\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-system-cni-dir\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.834074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlzd\" (UniqueName: \"kubernetes.io/projected/e39febc5-a0f0-4088-8f79-eb590108552a-kube-api-access-cwlzd\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-systemd\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-registration-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-device-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmckx\" (UniqueName: \"kubernetes.io/projected/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-kube-api-access-hmckx\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-kubernetes\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysctl-conf\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-run\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7549f02-e0b7-4791-acf5-cee526852cca-tmp\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-host\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ks8j\" (UniqueName: \"kubernetes.io/projected/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-kube-api-access-5ks8j\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a12dd506-758c-4108-a967-70c0c1220114-kubelet-config\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.834558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 21:08:03.835269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-host\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.835269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a12dd506-758c-4108-a967-70c0c1220114-dbus\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.835269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834672    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bs4hx\""
Sep 15 21:08:03.835269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.835269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-cnibin\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.835269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.834725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-socket-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.835652 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.835364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:03.835652 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:03.835430    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:03.837636 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.837617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:03.839559 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.839542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 21:08:03.839752 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.839716    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 21:08:03.839829 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.839787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sg7ng\""
Sep 15 21:08:03.871931 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.871906    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sjnb4"
Sep 15 21:08:03.879759 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.879729    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sjnb4"
Sep 15 21:08:03.901661 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.901602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal" event={"ID":"da7cf5090aadfc7dceb51f9b0a797cc1","Type":"ContainerStarted","Data":"68110e05b4619b86f678934eff5c15116f75af3edd0eed79dccb5769390b3ed9"}
Sep 15 21:08:03.903190 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.903166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal" event={"ID":"ab1af32928d1c75375f28ea688cf59c1","Type":"ContainerStarted","Data":"b74003f2737a1287ef782328cd0992e86a2aa638da72b6c11bb39e7cfa9b9b7d"}
Sep 15 21:08:03.922510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.922417    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-device-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-node-log\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj69q\" (UniqueName: \"kubernetes.io/projected/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-kube-api-access-mj69q\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-netns\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5jrz\" (UniqueName: \"kubernetes.io/projected/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-kube-api-access-c5jrz\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7549f02-e0b7-4791-acf5-cee526852cca-tmp\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-host\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ks8j\" (UniqueName: \"kubernetes.io/projected/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-kube-api-access-5ks8j\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-env-overrides\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovn-node-metrics-cert\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-cni-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-cni-bin\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-socket-dir-parent\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.935983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.935986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-multus-certs\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-etc-kubernetes\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfdt\" (UniqueName: \"kubernetes.io/projected/fe42b983-d82e-422e-8e6d-ce096a4992a6-kube-api-access-btfdt\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-host\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a12dd506-758c-4108-a967-70c0c1220114-dbus\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-etc-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-var-lib-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn88w\" (UniqueName: \"kubernetes.io/projected/52f78489-55d9-4e99-a0af-d2f73ca08ecf-kube-api-access-gn88w\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3e119ff-4a7f-4b5e-aafe-c48b610649af-agent-certs\") pod \"konnectivity-agent-xthx4\" (UID: \"a3e119ff-4a7f-4b5e-aafe-c48b610649af\") " pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysctl-d\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-var-lib-kubelet\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-sys-fs\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-serviceca\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-system-cni-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe42b983-d82e-422e-8e6d-ce096a4992a6-cni-binary-copy\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.936928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-k8s-cni-cncf-io\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlzd\" (UniqueName: \"kubernetes.io/projected/e39febc5-a0f0-4088-8f79-eb590108552a-kube-api-access-cwlzd\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f0d21f-f617-40af-998d-895e688399ce-tmp-dir\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovnkube-config\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-host-slash\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-systemd\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-registration-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmckx\" (UniqueName: \"kubernetes.io/projected/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-kube-api-access-hmckx\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-kubelet\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-daemon-config\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-ovn\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-kubernetes\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysctl-conf\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-run\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttv9\" (UniqueName: \"kubernetes.io/projected/00f0d21f-f617-40af-998d-895e688399ce-kube-api-access-kttv9\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:03.937740 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-kubelet\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-hostroot\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.936975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a12dd506-758c-4108-a967-70c0c1220114-kubelet-config\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00f0d21f-f617-40af-998d-895e688399ce-hosts-file\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-cni-multus\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-cnibin\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-socket-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-slash\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3e119ff-4a7f-4b5e-aafe-c48b610649af-konnectivity-ca\") pod \"konnectivity-agent-xthx4\" (UID: \"a3e119ff-4a7f-4b5e-aafe-c48b610649af\") " pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-sys\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-lib-modules\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9dd\" (UniqueName: \"kubernetes.io/projected/b7549f02-e0b7-4791-acf5-cee526852cca-kube-api-access-4l9dd\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-cni-bin\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.938519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-conf-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7549f02-e0b7-4791-acf5-cee526852cca-etc-tuned\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-run-netns\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-systemd\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-modprobe-d\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysconfig\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-system-cni-dir\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-systemd-units\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-device-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-log-socket\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovnkube-script-lib\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-iptables-alerter-script\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.937983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-cnibin\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-cni-netd\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-modprobe-d\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.939290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-host\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938107    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-host\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-run\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a12dd506-758c-4108-a967-70c0c1220114-kubelet-config\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysconfig\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:03.938726    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-system-cni-dir\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:03.938811    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret podName:a12dd506-758c-4108-a967-70c0c1220114 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:04.438788633 +0000 UTC m=+3.038700751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret") pod "global-pull-secret-syncer-tcvp4" (UID: "a12dd506-758c-4108-a967-70c0c1220114") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-var-lib-kubelet\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-lib-modules\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.938947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-sys-fs\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysctl-d\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-systemd\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-registration-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.940080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-serviceca\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e39febc5-a0f0-4088-8f79-eb590108552a-cnibin\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-kubernetes\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-socket-dir\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.939930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e39febc5-a0f0-4088-8f79-eb590108552a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.940151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-etc-sysctl-conf\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.940213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7549f02-e0b7-4791-acf5-cee526852cca-sys\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.940220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a12dd506-758c-4108-a967-70c0c1220114-dbus\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:03.941127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.940353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3e119ff-4a7f-4b5e-aafe-c48b610649af-konnectivity-ca\") pod \"konnectivity-agent-xthx4\" (UID: \"a3e119ff-4a7f-4b5e-aafe-c48b610649af\") " pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:03.942882 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.942853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7549f02-e0b7-4791-acf5-cee526852cca-tmp\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.942993 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.942887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7549f02-e0b7-4791-acf5-cee526852cca-etc-tuned\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.943054 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.943034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3e119ff-4a7f-4b5e-aafe-c48b610649af-agent-certs\") pod \"konnectivity-agent-xthx4\" (UID: \"a3e119ff-4a7f-4b5e-aafe-c48b610649af\") " pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:03.957698 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.957297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlzd\" (UniqueName: \"kubernetes.io/projected/e39febc5-a0f0-4088-8f79-eb590108552a-kube-api-access-cwlzd\") pod \"multus-additional-cni-plugins-7pdpq\" (UID: \"e39febc5-a0f0-4088-8f79-eb590108552a\") " pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:03.957698 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.957312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9dd\" (UniqueName: \"kubernetes.io/projected/b7549f02-e0b7-4791-acf5-cee526852cca-kube-api-access-4l9dd\") pod \"tuned-4jmnc\" (UID: \"b7549f02-e0b7-4791-acf5-cee526852cca\") " pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:03.957698 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.957299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmckx\" (UniqueName: \"kubernetes.io/projected/fd2a3b07-2f13-4c0e-b088-e55e12e803c8-kube-api-access-hmckx\") pod \"aws-ebs-csi-driver-node-hdq49\" (UID: \"fd2a3b07-2f13-4c0e-b088-e55e12e803c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:03.959473 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:03.959377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ks8j\" (UniqueName: \"kubernetes.io/projected/fd84df4b-1e3b-4784-a5b7-c2ea08ab8877-kube-api-access-5ks8j\") pod \"node-ca-lmfnh\" (UID: \"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877\") " pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:04.038434 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-systemd-units\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.038434 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-log-socket\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.038689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-log-socket\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.038689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-systemd-units\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.038689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovnkube-script-lib\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.038689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-iptables-alerter-script\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:04.038689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-cnibin\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.038689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-cni-netd\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.038689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.038689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-node-log\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj69q\" (UniqueName: \"kubernetes.io/projected/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-kube-api-access-mj69q\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-netns\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5jrz\" (UniqueName: \"kubernetes.io/projected/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-kube-api-access-c5jrz\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-env-overrides\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovn-node-metrics-cert\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-cni-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-cni-bin\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-socket-dir-parent\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-multus-certs\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-etc-kubernetes\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.038975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfdt\" (UniqueName: \"kubernetes.io/projected/fe42b983-d82e-422e-8e6d-ce096a4992a6-kube-api-access-btfdt\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-etc-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-var-lib-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn88w\" (UniqueName: \"kubernetes.io/projected/52f78489-55d9-4e99-a0af-d2f73ca08ecf-kube-api-access-gn88w\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-system-cni-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovnkube-script-lib\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe42b983-d82e-422e-8e6d-ce096a4992a6-cni-binary-copy\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-k8s-cni-cncf-io\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f0d21f-f617-40af-998d-895e688399ce-tmp-dir\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-cni-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovnkube-config\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-host-slash\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-kubelet\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-daemon-config\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-ovn\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttv9\" (UniqueName: \"kubernetes.io/projected/00f0d21f-f617-40af-998d-895e688399ce-kube-api-access-kttv9\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-kubelet\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-hostroot\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00f0d21f-f617-40af-998d-895e688399ce-hosts-file\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-cni-multus\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.039847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-slash\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-cni-bin\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-conf-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-run-netns\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-systemd\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-iptables-alerter-script\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-systemd\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-cnibin\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-cni-bin\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-cni-netd\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.039914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-node-log\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-netns\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-socket-dir-parent\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-multus-certs\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-etc-kubernetes\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.040708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-etc-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-run-ovn\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-kubelet\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-env-overrides\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-hostroot\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00f0d21f-f617-40af-998d-895e688399ce-hosts-file\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-cni-multus\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-slash\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.040995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-var-lib-cni-bin\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-daemon-config\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe42b983-d82e-422e-8e6d-ce096a4992a6-cni-binary-copy\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-host-run-k8s-cni-cncf-io\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-multus-conf-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-var-lib-openvswitch\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-run-netns\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe42b983-d82e-422e-8e6d-ce096a4992a6-system-cni-dir\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-host-slash\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:04.041530 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.041377    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:04.042173 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.041484    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:08:04.541430867 +0000 UTC m=+3.141342978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:04.042173 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52f78489-55d9-4e99-a0af-d2f73ca08ecf-host-kubelet\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.042173 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f0d21f-f617-40af-998d-895e688399ce-tmp-dir\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:04.042173 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.041690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovnkube-config\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.044435 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.044407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52f78489-55d9-4e99-a0af-d2f73ca08ecf-ovn-node-metrics-cert\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.045972 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.045944    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:04.046065 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.045983    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:04.046065 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.045998    2563 projected.go:196] Error preparing data for projected volume kube-api-access-r7b79 for pod openshift-network-diagnostics/network-check-target-w5n4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:04.046197 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.046156    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79 podName:b0527096-0133-4009-9e53-2b4c42629d64 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:04.546136541 +0000 UTC m=+3.146048649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r7b79" (UniqueName: "kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79") pod "network-check-target-w5n4g" (UID: "b0527096-0133-4009-9e53-2b4c42629d64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:04.050803 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.050744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5jrz\" (UniqueName: \"kubernetes.io/projected/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-kube-api-access-c5jrz\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:04.051638 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.051613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn88w\" (UniqueName: \"kubernetes.io/projected/52f78489-55d9-4e99-a0af-d2f73ca08ecf-kube-api-access-gn88w\") pod \"ovnkube-node-fxcmb\" (UID: \"52f78489-55d9-4e99-a0af-d2f73ca08ecf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.051847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.051823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj69q\" (UniqueName: \"kubernetes.io/projected/6c33483c-2ce4-484c-a2f7-ae2e64d616d1-kube-api-access-mj69q\") pod \"iptables-alerter-wbdhj\" (UID: \"6c33483c-2ce4-484c-a2f7-ae2e64d616d1\") " pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:04.052799 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.052774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttv9\" (UniqueName: \"kubernetes.io/projected/00f0d21f-f617-40af-998d-895e688399ce-kube-api-access-kttv9\") pod \"node-resolver-zngks\" (UID: \"00f0d21f-f617-40af-998d-895e688399ce\") " pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:04.053229 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.053208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfdt\" (UniqueName: \"kubernetes.io/projected/fe42b983-d82e-422e-8e6d-ce096a4992a6-kube-api-access-btfdt\") pod \"multus-xk55v\" (UID: \"fe42b983-d82e-422e-8e6d-ce096a4992a6\") " pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.122170 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.122130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:04.132150 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.132120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4jmnc"
Sep 15 21:08:04.141497 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:04.141460    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e119ff_4a7f_4b5e_aafe_c48b610649af.slice/crio-b7d3643474ba728b91cb38083ca954ee295c043119a33e3effacff09c9127add WatchSource:0}: Error finding container b7d3643474ba728b91cb38083ca954ee295c043119a33e3effacff09c9127add: Status 404 returned error can't find the container with id b7d3643474ba728b91cb38083ca954ee295c043119a33e3effacff09c9127add
Sep 15 21:08:04.143679 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.143600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7pdpq"
Sep 15 21:08:04.148864 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:04.148832    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7549f02_e0b7_4791_acf5_cee526852cca.slice/crio-f5d62e86faaced941b7f5820137151e03b7a2dfa925bcb5efa2fc394b00963a7 WatchSource:0}: Error finding container f5d62e86faaced941b7f5820137151e03b7a2dfa925bcb5efa2fc394b00963a7: Status 404 returned error can't find the container with id f5d62e86faaced941b7f5820137151e03b7a2dfa925bcb5efa2fc394b00963a7
Sep 15 21:08:04.149660 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.149640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49"
Sep 15 21:08:04.156554 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.156527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lmfnh"
Sep 15 21:08:04.156995 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:04.156958    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39febc5_a0f0_4088_8f79_eb590108552a.slice/crio-e2ec42914f906df2b95be6d2902424e853ae39f29972e6f7deafdc33a6478da9 WatchSource:0}: Error finding container e2ec42914f906df2b95be6d2902424e853ae39f29972e6f7deafdc33a6478da9: Status 404 returned error can't find the container with id e2ec42914f906df2b95be6d2902424e853ae39f29972e6f7deafdc33a6478da9
Sep 15 21:08:04.164736 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.164715    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:04.165029 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:04.165004    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2a3b07_2f13_4c0e_b088_e55e12e803c8.slice/crio-b21ce88c22e3fe2ec32374fd6afc5cb3c72227561ff32a4e3524d2f403d002b1 WatchSource:0}: Error finding container b21ce88c22e3fe2ec32374fd6afc5cb3c72227561ff32a4e3524d2f403d002b1: Status 404 returned error can't find the container with id b21ce88c22e3fe2ec32374fd6afc5cb3c72227561ff32a4e3524d2f403d002b1
Sep 15 21:08:04.167001 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:04.166966    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd84df4b_1e3b_4784_a5b7_c2ea08ab8877.slice/crio-0ce5e261e06eef351e31ed111dbd65d847a720e4989175e0d98e6add30d5dc7a WatchSource:0}: Error finding container 0ce5e261e06eef351e31ed111dbd65d847a720e4989175e0d98e6add30d5dc7a: Status 404 returned error can't find the container with id 0ce5e261e06eef351e31ed111dbd65d847a720e4989175e0d98e6add30d5dc7a
Sep 15 21:08:04.173371 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.173345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wbdhj"
Sep 15 21:08:04.177735 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:04.177706    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f78489_55d9_4e99_a0af_d2f73ca08ecf.slice/crio-eae41304a4f3e4a84b9b3f80892e3e83fc4fe43dd77b3ac0ecf3bdb037aa0e76 WatchSource:0}: Error finding container eae41304a4f3e4a84b9b3f80892e3e83fc4fe43dd77b3ac0ecf3bdb037aa0e76: Status 404 returned error can't find the container with id eae41304a4f3e4a84b9b3f80892e3e83fc4fe43dd77b3ac0ecf3bdb037aa0e76
Sep 15 21:08:04.179571 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.179527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xk55v"
Sep 15 21:08:04.182501 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:04.182476    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c33483c_2ce4_484c_a2f7_ae2e64d616d1.slice/crio-3c2a073cc706d0c5ca94f0c3fa38f3a6a9f90a51295fedceec36d7e01a8b396a WatchSource:0}: Error finding container 3c2a073cc706d0c5ca94f0c3fa38f3a6a9f90a51295fedceec36d7e01a8b396a: Status 404 returned error can't find the container with id 3c2a073cc706d0c5ca94f0c3fa38f3a6a9f90a51295fedceec36d7e01a8b396a
Sep 15 21:08:04.187528 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.187508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zngks"
Sep 15 21:08:04.199822 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:04.199787    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f0d21f_f617_40af_998d_895e688399ce.slice/crio-e7e084fb75bc66dac97f2b99258c8e162b7916d723a1be1749f88edb4457021d WatchSource:0}: Error finding container e7e084fb75bc66dac97f2b99258c8e162b7916d723a1be1749f88edb4457021d: Status 404 returned error can't find the container with id e7e084fb75bc66dac97f2b99258c8e162b7916d723a1be1749f88edb4457021d
Sep 15 21:08:04.443155 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.443061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:04.443331 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.443226    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:04.443331 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.443304    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret podName:a12dd506-758c-4108-a967-70c0c1220114 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:05.443286101 +0000 UTC m=+4.043198189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret") pod "global-pull-secret-syncer-tcvp4" (UID: "a12dd506-758c-4108-a967-70c0c1220114") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:04.543655 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.543616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:04.543830 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.543810    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:04.543900 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.543888    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:08:05.543869247 +0000 UTC m=+4.143781338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:04.644096 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.644061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:04.644267 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.644251    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:04.644326 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.644275    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:04.644326 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.644290    2563 projected.go:196] Error preparing data for projected volume kube-api-access-r7b79 for pod openshift-network-diagnostics/network-check-target-w5n4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:04.644410 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.644352    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79 podName:b0527096-0133-4009-9e53-2b4c42629d64 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:05.644337711 +0000 UTC m=+4.244249800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7b79" (UniqueName: "kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79") pod "network-check-target-w5n4g" (UID: "b0527096-0133-4009-9e53-2b4c42629d64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:04.882883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.882779    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:03:03 +0000 UTC" deadline="2028-02-14 15:32:21.092167617 +0000 UTC"
Sep 15 21:08:04.882883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.882823    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12402h24m16.209348458s"
Sep 15 21:08:04.895528 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.895308    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:04.895528 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:04.895461    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:04.907758 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.907709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wbdhj" event={"ID":"6c33483c-2ce4-484c-a2f7-ae2e64d616d1","Type":"ContainerStarted","Data":"3c2a073cc706d0c5ca94f0c3fa38f3a6a9f90a51295fedceec36d7e01a8b396a"}
Sep 15 21:08:04.910670 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.910638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"eae41304a4f3e4a84b9b3f80892e3e83fc4fe43dd77b3ac0ecf3bdb037aa0e76"}
Sep 15 21:08:04.913981 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.913951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lmfnh" event={"ID":"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877","Type":"ContainerStarted","Data":"0ce5e261e06eef351e31ed111dbd65d847a720e4989175e0d98e6add30d5dc7a"}
Sep 15 21:08:04.916302 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.916277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49" event={"ID":"fd2a3b07-2f13-4c0e-b088-e55e12e803c8","Type":"ContainerStarted","Data":"b21ce88c22e3fe2ec32374fd6afc5cb3c72227561ff32a4e3524d2f403d002b1"}
Sep 15 21:08:04.928573 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.918072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4jmnc" event={"ID":"b7549f02-e0b7-4791-acf5-cee526852cca","Type":"ContainerStarted","Data":"f5d62e86faaced941b7f5820137151e03b7a2dfa925bcb5efa2fc394b00963a7"}
Sep 15 21:08:04.928573 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.926040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zngks" event={"ID":"00f0d21f-f617-40af-998d-895e688399ce","Type":"ContainerStarted","Data":"e7e084fb75bc66dac97f2b99258c8e162b7916d723a1be1749f88edb4457021d"}
Sep 15 21:08:04.934223 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.934186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xk55v" event={"ID":"fe42b983-d82e-422e-8e6d-ce096a4992a6","Type":"ContainerStarted","Data":"e28463b6e4a0907d2b7b29b5e5fd6781795df0c247528b9128e5cd201bffbadd"}
Sep 15 21:08:04.938756 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.938728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerStarted","Data":"e2ec42914f906df2b95be6d2902424e853ae39f29972e6f7deafdc33a6478da9"}
Sep 15 21:08:04.943218 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:04.943189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xthx4" event={"ID":"a3e119ff-4a7f-4b5e-aafe-c48b610649af","Type":"ContainerStarted","Data":"b7d3643474ba728b91cb38083ca954ee295c043119a33e3effacff09c9127add"}
Sep 15 21:08:05.451326 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:05.451281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:05.451489 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.451475    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:05.451564 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.451558    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret podName:a12dd506-758c-4108-a967-70c0c1220114 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:07.451536641 +0000 UTC m=+6.051448748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret") pod "global-pull-secret-syncer-tcvp4" (UID: "a12dd506-758c-4108-a967-70c0c1220114") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:05.552538 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:05.552425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:05.552709 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.552641    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:05.552766 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.552710    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:08:07.552689725 +0000 UTC m=+6.152601819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:05.653533 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:05.653466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:05.653746 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.653676    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:05.653746 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.653698    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:05.653746 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.653710    2563 projected.go:196] Error preparing data for projected volume kube-api-access-r7b79 for pod openshift-network-diagnostics/network-check-target-w5n4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:05.653918 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.653770    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79 podName:b0527096-0133-4009-9e53-2b4c42629d64 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:07.653751986 +0000 UTC m=+6.253664088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7b79" (UniqueName: "kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79") pod "network-check-target-w5n4g" (UID: "b0527096-0133-4009-9e53-2b4c42629d64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:05.884015 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:05.883924    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:03:03 +0000 UTC" deadline="2028-03-05 21:35:41.935354992 +0000 UTC"
Sep 15 21:08:05.884015 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:05.883968    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12888h27m36.051390973s"
Sep 15 21:08:05.897532 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:05.895213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:05.897532 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:05.895235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:05.897532 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.895321    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:05.897532 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:05.895716    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:06.895862 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:06.895299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:06.895862 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:06.895413    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:06.961116 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:06.961075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal" event={"ID":"ab1af32928d1c75375f28ea688cf59c1","Type":"ContainerStarted","Data":"a0376385f2a5603990d13246163d5ff951ef8f9b143fc4d9296253c400a7c1be"}
Sep 15 21:08:06.976000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:06.975940    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-113.ec2.internal" podStartSLOduration=3.9759177660000002 podStartE2EDuration="3.975917766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:06.975611847 +0000 UTC m=+5.575523957" watchObservedRunningTime="2026-09-15 21:08:06.975917766 +0000 UTC m=+5.575829879"
Sep 15 21:08:07.473231 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:07.473128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:07.473380 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.473279    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:07.473380 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.473345    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret podName:a12dd506-758c-4108-a967-70c0c1220114 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:11.473326555 +0000 UTC m=+10.073238658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret") pod "global-pull-secret-syncer-tcvp4" (UID: "a12dd506-758c-4108-a967-70c0c1220114") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:07.573710 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:07.573674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:07.573888 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.573828    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:07.573888 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.573884    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:08:11.573869649 +0000 UTC m=+10.173781737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:07.674905 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:07.674863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:07.675160 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.675105    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:07.675160 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.675127    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:07.675160 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.675141    2563 projected.go:196] Error preparing data for projected volume kube-api-access-r7b79 for pod openshift-network-diagnostics/network-check-target-w5n4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:07.675311 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.675219    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79 podName:b0527096-0133-4009-9e53-2b4c42629d64 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:11.675201455 +0000 UTC m=+10.275113547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7b79" (UniqueName: "kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79") pod "network-check-target-w5n4g" (UID: "b0527096-0133-4009-9e53-2b4c42629d64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:07.895665 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:07.895571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:07.895870 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.895703    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:07.897464 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:07.897418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:07.897890 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:07.897556    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:07.970655 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:07.970580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal" event={"ID":"da7cf5090aadfc7dceb51f9b0a797cc1","Type":"ContainerStarted","Data":"e0b66abd2db6d44a0070fc155d8566f0c0cfd4f1ef14065e7d316d666495dfd4"}
Sep 15 21:08:08.894924 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:08.894887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:08.895202 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:08.895028    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:09.896310 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:09.896064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:09.896797 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:09.896419    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:09.897042 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:09.896886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:09.897042 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:09.897000    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:10.895103 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:10.895022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:10.895268 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:10.895157    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:11.507236 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:11.507195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:11.507708 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.507357    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:11.507708 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.507421    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret podName:a12dd506-758c-4108-a967-70c0c1220114 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:19.507402491 +0000 UTC m=+18.107314596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret") pod "global-pull-secret-syncer-tcvp4" (UID: "a12dd506-758c-4108-a967-70c0c1220114") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:11.608524 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:11.608488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:11.608695 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.608667    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:11.608773 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.608746    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:08:19.608723663 +0000 UTC m=+18.208635763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:11.709205 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:11.709168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:11.709373 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.709345    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:11.709484 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.709380    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:11.709484 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.709391    2563 projected.go:196] Error preparing data for projected volume kube-api-access-r7b79 for pod openshift-network-diagnostics/network-check-target-w5n4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:11.709484 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.709468    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79 podName:b0527096-0133-4009-9e53-2b4c42629d64 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:19.709433031 +0000 UTC m=+18.309345118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7b79" (UniqueName: "kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79") pod "network-check-target-w5n4g" (UID: "b0527096-0133-4009-9e53-2b4c42629d64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:11.894813 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:11.894731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:11.894980 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.894868    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:11.898467 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:11.896636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:11.898467 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:11.896749    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:12.894692 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:12.894585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:12.895080 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:12.894719    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:12.985254 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:12.985212    2563 generic.go:356] "Generic (PLEG): container finished" podID="da7cf5090aadfc7dceb51f9b0a797cc1" containerID="e0b66abd2db6d44a0070fc155d8566f0c0cfd4f1ef14065e7d316d666495dfd4" exitCode=0
Sep 15 21:08:12.985464 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:12.985260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal" event={"ID":"da7cf5090aadfc7dceb51f9b0a797cc1","Type":"ContainerDied","Data":"e0b66abd2db6d44a0070fc155d8566f0c0cfd4f1ef14065e7d316d666495dfd4"}
Sep 15 21:08:13.897237 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:13.897196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:13.897710 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:13.897325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:13.897802 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:13.897781    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:13.897913 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:13.897897    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:14.894787 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:14.894749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:14.894945 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:14.894902    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:15.898133 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:15.898096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:15.898615 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:15.898140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:15.898615 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:15.898230    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:15.898615 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:15.898352    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:16.894942 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:16.894912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:16.895134 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:16.895023    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:17.894379 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:17.894343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:17.894792 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:17.894481    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:17.894792 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:17.894538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:17.894792 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:17.894652    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:18.894675 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:18.894589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:18.895133 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:18.894724    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:19.569224 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:19.569169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:19.569533 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.569349    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:19.569533 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.569431    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret podName:a12dd506-758c-4108-a967-70c0c1220114 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:35.569411085 +0000 UTC m=+34.169323177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret") pod "global-pull-secret-syncer-tcvp4" (UID: "a12dd506-758c-4108-a967-70c0c1220114") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:19.670549 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:19.670504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:19.670716 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.670685    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:19.670785 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.670773    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:08:35.67075289 +0000 UTC m=+34.270664985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:19.770930 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:19.770890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:19.771094 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.771034    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:19.771094 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.771050    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:19.771094 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.771060    2563 projected.go:196] Error preparing data for projected volume kube-api-access-r7b79 for pod openshift-network-diagnostics/network-check-target-w5n4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:19.771196 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.771117    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79 podName:b0527096-0133-4009-9e53-2b4c42629d64 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:35.7711039 +0000 UTC m=+34.371015988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7b79" (UniqueName: "kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79") pod "network-check-target-w5n4g" (UID: "b0527096-0133-4009-9e53-2b4c42629d64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:19.898598 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:19.898504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:19.898598 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:19.898524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:19.899043 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.898628    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:19.899043 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:19.898769    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:20.895067 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:20.895034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:20.895261 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:20.895156    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:21.895763 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:21.895724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:21.896228 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:21.895826    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:21.896228 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:21.895877    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:21.896228 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:21.895915    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:22.895476 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:22.894983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:22.895476 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:22.895081    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:23.007700 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.006950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerStarted","Data":"061ea925ce0c53cd7bce486a378b7c90e979f7bdcd47c8af84ad1b6c70792fab"}
Sep 15 21:08:23.011105 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.010112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xthx4" event={"ID":"a3e119ff-4a7f-4b5e-aafe-c48b610649af","Type":"ContainerStarted","Data":"014d518dd9c996b2731a6cb43b0dc76050efb9927bc9257a8bdc931fe6e57e29"}
Sep 15 21:08:23.014931 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.013170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal" event={"ID":"da7cf5090aadfc7dceb51f9b0a797cc1","Type":"ContainerStarted","Data":"ec090459c0cf1c1e5312ea3dfffb3e5d33412518d628baf27aadff2247d7983f"}
Sep 15 21:08:23.016917 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.016894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"8612db3442f87f9bcf139d4534b092ea12edbc7724b8949285fe5914c94dc75a"}
Sep 15 21:08:23.019315 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.019147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lmfnh" event={"ID":"fd84df4b-1e3b-4784-a5b7-c2ea08ab8877","Type":"ContainerStarted","Data":"2282edea5fb0f3a9bf19ade79df0f092f9cae863223a0562a036a24b158ce634"}
Sep 15 21:08:23.023816 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.023794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49" event={"ID":"fd2a3b07-2f13-4c0e-b088-e55e12e803c8","Type":"ContainerStarted","Data":"58595eb369a236d96613051ea82adb72dc80014a770db816e005308d6388501c"}
Sep 15 21:08:23.025898 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.025877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4jmnc" event={"ID":"b7549f02-e0b7-4791-acf5-cee526852cca","Type":"ContainerStarted","Data":"98247639edca8736f839ed380e494c0f9d92952f413096dc8c7ccfa4203fcba9"}
Sep 15 21:08:23.044092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.044038    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zngks" podStartSLOduration=2.566019597 podStartE2EDuration="21.044020827s" totalImagesPullingTime="18.47800123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:23.043545011 +0000 UTC m=+21.643457123" watchObservedRunningTime="2026-09-15 21:08:23.044020827 +0000 UTC m=+21.643932938"
Sep 15 21:08:23.062918 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.062740    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-113.ec2.internal" podStartSLOduration=20.062723332 podStartE2EDuration="20.062723332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:23.062527189 +0000 UTC m=+21.662439299" watchObservedRunningTime="2026-09-15 21:08:23.062723332 +0000 UTC m=+21.662635443"
Sep 15 21:08:23.076894 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.076854    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xthx4" podStartSLOduration=6.757856197 podStartE2EDuration="21.076840176s" totalImagesPullingTime="14.318983979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:23.076576016 +0000 UTC m=+21.676488127" watchObservedRunningTime="2026-09-15 21:08:23.076840176 +0000 UTC m=+21.676752286"
Sep 15 21:08:23.094553 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.093603    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4jmnc" podStartSLOduration=2.563807224 podStartE2EDuration="21.093589376s" totalImagesPullingTime="18.529782152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:23.093342727 +0000 UTC m=+21.693254837" watchObservedRunningTime="2026-09-15 21:08:23.093589376 +0000 UTC m=+21.693501486"
Sep 15 21:08:23.108395 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.108356    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lmfnh" podStartSLOduration=2.6416558500000002 podStartE2EDuration="21.108343981s" totalImagesPullingTime="18.466688131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:23.108218329 +0000 UTC m=+21.708130441" watchObservedRunningTime="2026-09-15 21:08:23.108343981 +0000 UTC m=+21.708256092"
Sep 15 21:08:23.898140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.897716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:23.898140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:23.897731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:23.898140 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:23.897833    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:23.898140 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:23.897928    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:24.031510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.031314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zngks" event={"ID":"00f0d21f-f617-40af-998d-895e688399ce","Type":"ContainerStarted","Data":"c1e191ce716b493ed5f0eb971b64ca1a0f9485ab49d6872a608fb58e47035633"}
Sep 15 21:08:24.033896 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.033848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xk55v" event={"ID":"fe42b983-d82e-422e-8e6d-ce096a4992a6","Type":"ContainerStarted","Data":"6945c51d1fda54a4f42426c6f87641bc696f4ddf315252549fdd6ce4f17ea502"}
Sep 15 21:08:24.036195 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.036169    2563 generic.go:356] "Generic (PLEG): container finished" podID="e39febc5-a0f0-4088-8f79-eb590108552a" containerID="061ea925ce0c53cd7bce486a378b7c90e979f7bdcd47c8af84ad1b6c70792fab" exitCode=0
Sep 15 21:08:24.036304 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.036253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerDied","Data":"061ea925ce0c53cd7bce486a378b7c90e979f7bdcd47c8af84ad1b6c70792fab"}
Sep 15 21:08:24.040712 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.038811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wbdhj" event={"ID":"6c33483c-2ce4-484c-a2f7-ae2e64d616d1","Type":"ContainerStarted","Data":"06f0d929e958e8b5f32f5a6904b306a34916830a215ca899bc850162f6047a44"}
Sep 15 21:08:24.044167 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.044110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"2027e19786f811cd8906c7c652d38a6bdd73ae04441e2373e4518cc2822404d8"}
Sep 15 21:08:24.044243 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.044196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"7c26b802c33fb89fa90dbf995f80a9b0d920a5162ad4750cc430947e6a1a0778"}
Sep 15 21:08:24.044288 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.044247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"b1ac0e894d5b2ebffe656c67df6cb2fceb7efc4b1af9216addb1c5473d4a5bac"}
Sep 15 21:08:24.044334 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.044299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"29112ff723fa776c6c56472957b1a5676ba5ebaf19026610ed83347bfc437e8c"}
Sep 15 21:08:24.044384 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.044351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"17b1214c0c7332eb5abd05d09e3ec4ec46b1ff85ddd576a2573cc96fde9fa990"}
Sep 15 21:08:24.050817 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.050780    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xk55v" podStartSLOduration=3.245771597 podStartE2EDuration="22.050768101s" totalImagesPullingTime="18.804996504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:24.05022772 +0000 UTC m=+22.650139832" watchObservedRunningTime="2026-09-15 21:08:24.050768101 +0000 UTC m=+22.650680210"
Sep 15 21:08:24.079612 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.079564    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wbdhj" podStartSLOduration=3.628849239 podStartE2EDuration="22.07955218s" totalImagesPullingTime="18.450702941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:24.079536301 +0000 UTC m=+22.679448412" watchObservedRunningTime="2026-09-15 21:08:24.07955218 +0000 UTC m=+22.679464289"
Sep 15 21:08:24.244510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.244343    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 21:08:24.271142 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.271118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:24.271599 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.271582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:24.894773 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.894744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:24.894960 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:24.894859    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:24.902970 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.902849    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:08:24.24450568Z","UUID":"d4d83b43-f2f3-4bf5-8537-e439db67d582","Handler":null,"Name":"","Endpoint":""}
Sep 15 21:08:24.904588 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.904557    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 21:08:24.904588 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:24.904591    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 21:08:25.049386 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:25.049350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49" event={"ID":"fd2a3b07-2f13-4c0e-b088-e55e12e803c8","Type":"ContainerStarted","Data":"78484ce36bdee4f7c9b0d7c356af8a76964898aa750afb1de5649112db9ad38d"}
Sep 15 21:08:25.049797 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:25.049711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:25.050139 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:25.050113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xthx4"
Sep 15 21:08:25.895417 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:25.895377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:25.895615 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:25.895377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:25.895615 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:25.895549    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:25.895733 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:25.895696    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:26.055258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:26.055222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"a3f9c0820871d7d73da4880ae167100f22ed9d7b95136b617d3a791131aa934b"}
Sep 15 21:08:26.057656 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:26.057625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49" event={"ID":"fd2a3b07-2f13-4c0e-b088-e55e12e803c8","Type":"ContainerStarted","Data":"a4f1fcd4ed4ea3c7bf8ccd9fdc7a250862531fadafb1e75d012aec5c69937372"}
Sep 15 21:08:26.076431 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:26.076386    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdq49" podStartSLOduration=3.251720146 podStartE2EDuration="24.076372887s" totalImagesPullingTime="20.824652741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:26.076166619 +0000 UTC m=+24.676078730" watchObservedRunningTime="2026-09-15 21:08:26.076372887 +0000 UTC m=+24.676284997"
Sep 15 21:08:26.895299 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:26.895269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:26.895498 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:26.895384    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:27.894636 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:27.894545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:27.895031 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:27.894548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:27.895031 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:27.894682    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:27.895031 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:27.894784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:28.894742 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:28.894557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:28.895209 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:28.894821    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:29.071953 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.071861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" event={"ID":"52f78489-55d9-4e99-a0af-d2f73ca08ecf","Type":"ContainerStarted","Data":"b53c22b4d48400a23494519ce44ccf2fbadc18990c8164a048fb065e71631512"}
Sep 15 21:08:29.072137 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.072107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:29.072258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.072183    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:29.074003 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.073968    2563 generic.go:356] "Generic (PLEG): container finished" podID="e39febc5-a0f0-4088-8f79-eb590108552a" containerID="29d496cea1643ce9385ac1512bd42500b918d2afc8f9a07847f115aed3eaa57c" exitCode=0
Sep 15 21:08:29.074083 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.074004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerDied","Data":"29d496cea1643ce9385ac1512bd42500b918d2afc8f9a07847f115aed3eaa57c"}
Sep 15 21:08:29.088944 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.088925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:29.089035 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.089010    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:29.103217 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.103182    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb" podStartSLOduration=8.584167009 podStartE2EDuration="27.103170557s" totalImagesPullingTime="18.519003548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:29.103034514 +0000 UTC m=+27.702946624" watchObservedRunningTime="2026-09-15 21:08:29.103170557 +0000 UTC m=+27.703082667"
Sep 15 21:08:29.899337 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.899043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:29.899337 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:29.899179    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:29.899337 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:29.899215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:29.899337 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:29.899303    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:30.081099 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:30.081068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerStarted","Data":"9304a497c805b27df2d755d3a72e2bda20d77bed484df9ed7a4efc6eab78eab3"}
Sep 15 21:08:30.081263 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:30.081089    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 21:08:30.155937 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:30.155867    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tcvp4"]
Sep 15 21:08:30.156066 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:30.156009    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:30.156152 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:30.156128    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:30.158006 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:30.157982    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w5n4g"]
Sep 15 21:08:30.158106 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:30.158070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:30.158183 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:30.158160    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:30.161809 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:30.161788    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jlctj"]
Sep 15 21:08:30.161892 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:30.161873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:30.161966 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:30.161951    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:31.085549 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:31.085503    2563 generic.go:356] "Generic (PLEG): container finished" podID="e39febc5-a0f0-4088-8f79-eb590108552a" containerID="9304a497c805b27df2d755d3a72e2bda20d77bed484df9ed7a4efc6eab78eab3" exitCode=0
Sep 15 21:08:31.086106 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:31.085586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerDied","Data":"9304a497c805b27df2d755d3a72e2bda20d77bed484df9ed7a4efc6eab78eab3"}
Sep 15 21:08:31.086106 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:31.085737    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 21:08:31.895314 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:31.895152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:31.895415 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:31.895377    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:31.895415 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:31.895237    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:31.895542 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:31.895221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:31.895542 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:31.895460    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:31.895638 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:31.895584    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:31.992318 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:31.992283    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:08:32.093017 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:32.092915    2563 generic.go:356] "Generic (PLEG): container finished" podID="e39febc5-a0f0-4088-8f79-eb590108552a" containerID="c911e9c50f726914a5e2a40ee85ba17a0c74bb4d30d946933139f9c6d3242ca4" exitCode=0
Sep 15 21:08:32.093017 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:32.092999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerDied","Data":"c911e9c50f726914a5e2a40ee85ba17a0c74bb4d30d946933139f9c6d3242ca4"}
Sep 15 21:08:33.894564 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:33.894497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:33.894564 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:33.894507    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:33.895029 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:33.894595    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5n4g" podUID="b0527096-0133-4009-9e53-2b4c42629d64"
Sep 15 21:08:33.895029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:33.894601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:33.895029 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:33.894713    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:08:33.895029 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:33.894820    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tcvp4" podUID="a12dd506-758c-4108-a967-70c0c1220114"
Sep 15 21:08:35.587420 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.587385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:35.587788 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.587594    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:35.587788 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.587676    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret podName:a12dd506-758c-4108-a967-70c0c1220114 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:07.587655736 +0000 UTC m=+66.187567830 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret") pod "global-pull-secret-syncer-tcvp4" (UID: "a12dd506-758c-4108-a967-70c0c1220114") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:35.664365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.664338    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-113.ec2.internal" event="NodeReady"
Sep 15 21:08:35.664542 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.664510    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 21:08:35.688535 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.688465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:35.688681 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.688589    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:35.688681 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.688648    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:09:07.688633177 +0000 UTC m=+66.288545268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:35.705238 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.705207    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75f6f99d9-4wvm2"]
Sep 15 21:08:35.721778 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.721752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.723431 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.723404    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dlqdm"]
Sep 15 21:08:35.724384 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.724362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 21:08:35.724514 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.724372    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 21:08:35.724514 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.724464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 21:08:35.730762 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.730153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4hkpc\""
Sep 15 21:08:35.731139 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.731117    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 21:08:35.738744 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.737934    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75f6f99d9-4wvm2"]
Sep 15 21:08:35.738744 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.737972    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kv4ww"]
Sep 15 21:08:35.738744 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.738064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.740147 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.740125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 21:08:35.740249 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.740212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p5jgl\""
Sep 15 21:08:35.740306 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.740212    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 21:08:35.756240 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.756217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dlqdm"]
Sep 15 21:08:35.756358 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.756268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kv4ww"]
Sep 15 21:08:35.756422 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.756356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kv4ww"
Sep 15 21:08:35.758699 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.758681    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 21:08:35.758809 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.758742    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lthts\""
Sep 15 21:08:35.758809 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.758796    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 21:08:35.758919 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.758744    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 21:08:35.789051 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.789014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:35.789209 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.789195    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:35.789282 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.789246    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:35.789282 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.789261    2563 projected.go:196] Error preparing data for projected volume kube-api-access-r7b79 for pod openshift-network-diagnostics/network-check-target-w5n4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:35.789383 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.789328    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79 podName:b0527096-0133-4009-9e53-2b4c42629d64 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:07.789309521 +0000 UTC m=+66.389221610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7b79" (UniqueName: "kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79") pod "network-check-target-w5n4g" (UID: "b0527096-0133-4009-9e53-2b4c42629d64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:35.890381 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.890381 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpfx\" (UniqueName: \"kubernetes.io/projected/1fc041fc-3382-4159-b3be-8145231efc81-kube-api-access-zlpfx\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.890644 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dn9\" (UniqueName: \"kubernetes.io/projected/83cfd1f3-a4dc-4534-9151-0640d5a67623-kube-api-access-z8dn9\") pod \"ingress-canary-kv4ww\" (UID: \"83cfd1f3-a4dc-4534-9151-0640d5a67623\") " pod="openshift-ingress-canary/ingress-canary-kv4ww"
Sep 15 21:08:35.890644 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-trusted-ca\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.890644 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-installation-pull-secrets\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.890644 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc041fc-3382-4159-b3be-8145231efc81-tmp-dir\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.890644 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-image-registry-private-configuration\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.890883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-ca-trust-extracted\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.890883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc041fc-3382-4159-b3be-8145231efc81-config-volume\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.890883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-bound-sa-token\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.890883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9hs\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-kube-api-access-bp9hs\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.890883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.890883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-certificates\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.891126 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.890924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83cfd1f3-a4dc-4534-9151-0640d5a67623-cert\") pod \"ingress-canary-kv4ww\" (UID: \"83cfd1f3-a4dc-4534-9151-0640d5a67623\") " pod="openshift-ingress-canary/ingress-canary-kv4ww"
Sep 15 21:08:35.894405 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.894386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:08:35.894531 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.894386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:08:35.894592 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.894385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:08:35.896716 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.896695    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:08:35.896879 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.896697    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:08:35.897116 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.897096    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bkm5f\""
Sep 15 21:08:35.897204 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.897132    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lj4g\""
Sep 15 21:08:35.897433 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.897414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:08:35.897555 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.897464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:08:35.991906 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.991877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9hs\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-kube-api-access-bp9hs\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.992095 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.991923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.992095 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.991949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-certificates\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.992095 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.992058    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:08:35.992095 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.992078    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:08:35.992272 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.992129    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:08:36.492109709 +0000 UTC m=+35.092021814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:08:35.992272 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83cfd1f3-a4dc-4534-9151-0640d5a67623-cert\") pod \"ingress-canary-kv4ww\" (UID: \"83cfd1f3-a4dc-4534-9151-0640d5a67623\") " pod="openshift-ingress-canary/ingress-canary-kv4ww"
Sep 15 21:08:35.992272 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.992272 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpfx\" (UniqueName: \"kubernetes.io/projected/1fc041fc-3382-4159-b3be-8145231efc81-kube-api-access-zlpfx\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.992272 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dn9\" (UniqueName: \"kubernetes.io/projected/83cfd1f3-a4dc-4534-9151-0640d5a67623-kube-api-access-z8dn9\") pod \"ingress-canary-kv4ww\" (UID: \"83cfd1f3-a4dc-4534-9151-0640d5a67623\") " pod="openshift-ingress-canary/ingress-canary-kv4ww"
Sep 15 21:08:35.992423 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-trusted-ca\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.992423 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.992308    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:35.992423 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-installation-pull-secrets\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.992423 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:35.992359    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:36.492344217 +0000 UTC m=+35.092256322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:08:35.992423 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc041fc-3382-4159-b3be-8145231efc81-tmp-dir\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.992939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-image-registry-private-configuration\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.993043 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.992967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-ca-trust-extracted\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.993043 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.993009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc041fc-3382-4159-b3be-8145231efc81-config-volume\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.993111 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.993047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-bound-sa-token\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.993179 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.993160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-certificates\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.993999 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.993972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc041fc-3382-4159-b3be-8145231efc81-tmp-dir\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.994984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.994745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-trusted-ca\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.995207 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.995183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-ca-trust-extracted\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:35.998021 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.997821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc041fc-3382-4159-b3be-8145231efc81-config-volume\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:35.998125 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.998098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83cfd1f3-a4dc-4534-9151-0640d5a67623-cert\") pod \"ingress-canary-kv4ww\" (UID: \"83cfd1f3-a4dc-4534-9151-0640d5a67623\") " pod="openshift-ingress-canary/ingress-canary-kv4ww"
Sep 15 21:08:35.998617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:35.998528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-image-registry-private-configuration\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:36.001579 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.001554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9hs\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-kube-api-access-bp9hs\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:36.001697 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.001679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpfx\" (UniqueName: \"kubernetes.io/projected/1fc041fc-3382-4159-b3be-8145231efc81-kube-api-access-zlpfx\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:36.002140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.002122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dn9\" (UniqueName: \"kubernetes.io/projected/83cfd1f3-a4dc-4534-9151-0640d5a67623-kube-api-access-z8dn9\") pod \"ingress-canary-kv4ww\" (UID: \"83cfd1f3-a4dc-4534-9151-0640d5a67623\") " pod="openshift-ingress-canary/ingress-canary-kv4ww"
Sep 15 21:08:36.002354 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.002330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-bound-sa-token\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:36.002982 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.002964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-installation-pull-secrets\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:36.066315 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.066277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kv4ww"
Sep 15 21:08:36.391910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.391830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:36.482726 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.482689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:36.497597 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.497567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:36.497759 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:36.497639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:36.497823 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:36.497760    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:08:36.497823 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:36.497765    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:36.497823 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:36.497779    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:08:36.497995 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:36.497842    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:37.497823062 +0000 UTC m=+36.097735171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:08:36.497995 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:36.497862    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:08:37.497852337 +0000 UTC m=+36.097764431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:08:37.425753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:37.425485    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kv4ww"]
Sep 15 21:08:37.428291 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:08:37.428257    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cfd1f3_a4dc_4534_9151_0640d5a67623.slice/crio-57cb3d7eb84ca43c0842ba399edb59af5b256492997fd705f803cd8d6d29beb4 WatchSource:0}: Error finding container 57cb3d7eb84ca43c0842ba399edb59af5b256492997fd705f803cd8d6d29beb4: Status 404 returned error can't find the container with id 57cb3d7eb84ca43c0842ba399edb59af5b256492997fd705f803cd8d6d29beb4
Sep 15 21:08:37.505497 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:37.505470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:37.505602 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:37.505564    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:37.505686 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:37.505622    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:39.505604843 +0000 UTC m=+38.105516951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:08:37.505686 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:37.505564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:37.505764 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:37.505629    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:08:37.505764 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:37.505719    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:08:37.505829 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:37.505791    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:08:39.505772247 +0000 UTC m=+38.105684338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:08:38.109397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:38.109359    2563 generic.go:356] "Generic (PLEG): container finished" podID="e39febc5-a0f0-4088-8f79-eb590108552a" containerID="b11b3f09bdd8b2b5e624a34789ee24366733f84843d8f6a98a201a5ae82cfb38" exitCode=0
Sep 15 21:08:38.109578 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:38.109430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerDied","Data":"b11b3f09bdd8b2b5e624a34789ee24366733f84843d8f6a98a201a5ae82cfb38"}
Sep 15 21:08:38.111035 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:38.111005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kv4ww" event={"ID":"83cfd1f3-a4dc-4534-9151-0640d5a67623","Type":"ContainerStarted","Data":"57cb3d7eb84ca43c0842ba399edb59af5b256492997fd705f803cd8d6d29beb4"}
Sep 15 21:08:39.117219 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:39.117182    2563 generic.go:356] "Generic (PLEG): container finished" podID="e39febc5-a0f0-4088-8f79-eb590108552a" containerID="cf2f4f5da571adee051ceedce12772aa47cfdc54f10ada3648980a64e257ea29" exitCode=0
Sep 15 21:08:39.117853 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:39.117249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerDied","Data":"cf2f4f5da571adee051ceedce12772aa47cfdc54f10ada3648980a64e257ea29"}
Sep 15 21:08:39.520213 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:39.520175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:39.520400 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:39.520258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:39.520400 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:39.520352    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:08:39.520400 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:39.520376    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:08:39.520400 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:39.520380    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:39.520626 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:39.520439    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:43.520420544 +0000 UTC m=+42.120332632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:08:39.520626 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:39.520474    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:08:43.520464073 +0000 UTC m=+42.120376164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:08:40.122299 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:40.121685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kv4ww" event={"ID":"83cfd1f3-a4dc-4534-9151-0640d5a67623","Type":"ContainerStarted","Data":"9750d94bc4acfa0fa9305b04e2678c40d058ed7bb5970658cfd98d7755e6bb02"}
Sep 15 21:08:40.126574 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:40.126546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" event={"ID":"e39febc5-a0f0-4088-8f79-eb590108552a","Type":"ContainerStarted","Data":"c04577649184fb078bef805d28d54cee53e3c55944ab5c6b08088209d0959d56"}
Sep 15 21:08:40.136780 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:40.136733    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kv4ww" podStartSLOduration=2.615264196 podStartE2EDuration="5.136714385s" totalImagesPullingTime="2.521450189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:40.135779687 +0000 UTC m=+38.735691798" watchObservedRunningTime="2026-09-15 21:08:40.136714385 +0000 UTC m=+38.736626497"
Sep 15 21:08:40.155723 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:40.155679    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7pdpq" podStartSLOduration=8.300439394 podStartE2EDuration="38.155664686s" totalImagesPullingTime="29.855225292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:40.155312179 +0000 UTC m=+38.755224289" watchObservedRunningTime="2026-09-15 21:08:40.155664686 +0000 UTC m=+38.755576795"
Sep 15 21:08:40.279058 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:40.279008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:40.751561 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:40.751527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:41.056430 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:41.056352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:41.157631 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:41.157599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:43.550576 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:43.550536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:43.550964 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:43.550600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:43.550964 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:43.550689    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:08:43.550964 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:43.550692    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:43.550964 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:43.550753    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:51.550738988 +0000 UTC m=+50.150651079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:08:43.550964 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:43.550701    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:08:43.550964 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:43.550820    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:08:51.550806713 +0000 UTC m=+50.150718805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:08:43.911026 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:43.910932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:44.012593 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:44.012566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:51.609536 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:51.609501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:08:51.609998 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:08:51.609595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:08:51.609998 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:51.609660    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:08:51.609998 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:51.609687    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:08:51.609998 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:51.609739    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:51.609998 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:51.609747    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:09:07.609731826 +0000 UTC m=+66.209643914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:08:51.609998 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:08:51.609795    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:07.60978252 +0000 UTC m=+66.209694610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:09:02.103436 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:02.103410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxcmb"
Sep 15 21:09:07.622605 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.622567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:09:07.623124 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.622637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:09:07.623124 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.622666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:09:07.623124 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:07.622702    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:09:07.623124 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:07.622778    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:39.622762802 +0000 UTC m=+98.222674890 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:09:07.623124 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:07.622834    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:09:07.623124 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:07.622849    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:09:07.623124 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:07.622907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:09:39.622891381 +0000 UTC m=+98.222803483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:09:07.624988 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.624971    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:09:07.636083 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.636057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a12dd506-758c-4108-a967-70c0c1220114-original-pull-secret\") pod \"global-pull-secret-syncer-tcvp4\" (UID: \"a12dd506-758c-4108-a967-70c0c1220114\") " pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:09:07.713709 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.713685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcvp4"
Sep 15 21:09:07.723271 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.723246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:09:07.725298 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.725275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:09:07.733787 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:07.733769    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 21:09:07.733865 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:07.733846    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:10:11.733827011 +0000 UTC m=+130.333739099 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : secret "metrics-daemon-secret" not found
Sep 15 21:09:07.824239 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.824212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:09:07.826703 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.826685    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:09:07.837455 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.837424    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:09:07.848342 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:07.848325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7b79\" (UniqueName: \"kubernetes.io/projected/b0527096-0133-4009-9e53-2b4c42629d64-kube-api-access-r7b79\") pod \"network-check-target-w5n4g\" (UID: \"b0527096-0133-4009-9e53-2b4c42629d64\") " pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:09:08.008016 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:08.007988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bkm5f\""
Sep 15 21:09:08.016508 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:08.016486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tcvp4"]
Sep 15 21:09:08.016636 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:08.016488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:09:08.020044 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:09:08.020020    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12dd506_758c_4108_a967_70c0c1220114.slice/crio-96d352892477b811b02c788cea57a018941a6fa600fd59b570b114c570962efb WatchSource:0}: Error finding container 96d352892477b811b02c788cea57a018941a6fa600fd59b570b114c570962efb: Status 404 returned error can't find the container with id 96d352892477b811b02c788cea57a018941a6fa600fd59b570b114c570962efb
Sep 15 21:09:08.194324 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:08.194292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tcvp4" event={"ID":"a12dd506-758c-4108-a967-70c0c1220114","Type":"ContainerStarted","Data":"96d352892477b811b02c788cea57a018941a6fa600fd59b570b114c570962efb"}
Sep 15 21:09:08.311575 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:08.311508    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w5n4g"]
Sep 15 21:09:08.315260 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:09:08.315236    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0527096_0133_4009_9e53_2b4c42629d64.slice/crio-98c829de6a1135a5ab76c3d715f72b33c2013cb5023c8158837403b725a1f3f9 WatchSource:0}: Error finding container 98c829de6a1135a5ab76c3d715f72b33c2013cb5023c8158837403b725a1f3f9: Status 404 returned error can't find the container with id 98c829de6a1135a5ab76c3d715f72b33c2013cb5023c8158837403b725a1f3f9
Sep 15 21:09:09.198626 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:09.198554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w5n4g" event={"ID":"b0527096-0133-4009-9e53-2b4c42629d64","Type":"ContainerStarted","Data":"98c829de6a1135a5ab76c3d715f72b33c2013cb5023c8158837403b725a1f3f9"}
Sep 15 21:09:13.211668 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:13.211341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tcvp4" event={"ID":"a12dd506-758c-4108-a967-70c0c1220114","Type":"ContainerStarted","Data":"f324d8e9df361590dd04b6c35ed1ffc01d81d5befdff334754c5d0380e6fd963"}
Sep 15 21:09:13.215438 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:13.215406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w5n4g" event={"ID":"b0527096-0133-4009-9e53-2b4c42629d64","Type":"ContainerStarted","Data":"7605888c0f61bd915ffdb42306ce22eb98c322e52bdf6e112b151a9d1d7aafec"}
Sep 15 21:09:13.215618 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:13.215601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:09:13.249055 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:13.249014    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tcvp4" podStartSLOduration=66.458152986 podStartE2EDuration="1m11.248999472s" totalImagesPullingTime="4.790846486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:13.231063331 +0000 UTC m=+71.830975453" watchObservedRunningTime="2026-09-15 21:09:13.248999472 +0000 UTC m=+71.848911582"
Sep 15 21:09:13.249174 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:13.249127    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w5n4g" podStartSLOduration=66.757685945 podStartE2EDuration="1m11.249122577s" totalImagesPullingTime="4.491436632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:13.247545412 +0000 UTC m=+71.847457522" watchObservedRunningTime="2026-09-15 21:09:13.249122577 +0000 UTC m=+71.849034683"
Sep 15 21:09:39.641404 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:39.641250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:09:39.641404 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:39.641314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:09:39.641404 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:39.641407    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:09:39.642047 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:39.641414    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:09:39.642047 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:39.641420    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:09:39.642047 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:39.641515    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:43.641494134 +0000 UTC m=+162.241406222 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:09:39.642047 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:09:39.641554    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:10:43.641540739 +0000 UTC m=+162.241452840 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:09:44.219318 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:09:44.219292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w5n4g"
Sep 15 21:10:11.752243 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:10:11.752192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:10:11.752729 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:11.752310    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 21:10:11.752729 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:11.752374    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:12:13.752356677 +0000 UTC m=+252.352268764 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : secret "metrics-daemon-secret" not found
Sep 15 21:10:38.735614 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:38.735564    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" podUID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"
Sep 15 21:10:38.750440 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:38.750415    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dlqdm" podUID="1fc041fc-3382-4159-b3be-8145231efc81"
Sep 15 21:10:38.919384 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:38.919341    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:10:39.412685 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:10:39.412654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:10:39.412874 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:10:39.412654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:10:43.678894 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:10:43.678831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:10:43.678894 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:10:43.678901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:10:43.679406 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:43.678988    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:10:43.679406 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:43.679008    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:10:43.679406 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:43.679011    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6f99d9-4wvm2: secret "image-registry-tls" not found
Sep 15 21:10:43.679406 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:43.679063    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls podName:1fc041fc-3382-4159-b3be-8145231efc81 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:45.679048029 +0000 UTC m=+284.278960120 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls") pod "dns-default-dlqdm" (UID: "1fc041fc-3382-4159-b3be-8145231efc81") : secret "dns-default-metrics-tls" not found
Sep 15 21:10:43.679406 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:10:43.679102    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls podName:f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf nodeName:}" failed. No retries permitted until 2026-09-15 21:12:45.679087479 +0000 UTC m=+284.278999588 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls") pod "image-registry-75f6f99d9-4wvm2" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf") : secret "image-registry-tls" not found
Sep 15 21:10:53.894630 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:10:53.894544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:12:13.770919 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:13.770867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:12:13.771391 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:13.770995    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 21:12:13.771391 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:13.771063    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs podName:eb9743c9-fe1a-4518-83aa-d84dbabcac9f nodeName:}" failed. No retries permitted until 2026-09-15 21:14:15.771049246 +0000 UTC m=+374.370961334 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs") pod "network-metrics-daemon-jlctj" (UID: "eb9743c9-fe1a-4518-83aa-d84dbabcac9f") : secret "metrics-daemon-secret" not found
Sep 15 21:12:18.698554 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.698515    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"]
Sep 15 21:12:18.701439 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.701417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.703682 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.703654    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"]
Sep 15 21:12:18.704439 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.704414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 21:12:18.704566 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.704416    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 21:12:18.706360 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.706337    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"]
Sep 15 21:12:18.707387 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.706843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.708104 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.708067    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jr42n\""
Sep 15 21:12:18.709125 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.709101    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 15 21:12:18.709514 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.709496    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 15 21:12:18.710868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.710836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:18.711756 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.711736    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 21:12:18.712140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.712127    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 21:12:18.712192 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.712126    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 21:12:18.712848 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.712832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6gx7v\""
Sep 15 21:12:18.712848 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.712842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:12:18.712955 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.712855    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:12:18.713066 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.713052    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 15 21:12:18.713406 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.713394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9cgzx\""
Sep 15 21:12:18.713647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.713630    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 15 21:12:18.722069 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.722047    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"]
Sep 15 21:12:18.723260 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.723241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"]
Sep 15 21:12:18.727534 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.727515    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"]
Sep 15 21:12:18.791824 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.791791    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb"]
Sep 15 21:12:18.794628 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.794612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb"
Sep 15 21:12:18.796583 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.796556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dfjtr\""
Sep 15 21:12:18.799904 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.799883    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx"]
Sep 15 21:12:18.802629 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.802611    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"]
Sep 15 21:12:18.802742 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.802725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx"
Sep 15 21:12:18.805352 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.805479 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a17472-2d2b-45ac-98c1-cce356627d5c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.805479 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlpp8\" (UniqueName: \"kubernetes.io/projected/90788b65-78a3-47a7-b1e7-78c3c55c3033-kube-api-access-wlpp8\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:18.805587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a17472-2d2b-45ac-98c1-cce356627d5c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.805587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805506    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb"]
Sep 15 21:12:18.805587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s79m9\" (UniqueName: \"kubernetes.io/projected/99a17472-2d2b-45ac-98c1-cce356627d5c-kube-api-access-s79m9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.805587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmxks\" (UniqueName: \"kubernetes.io/projected/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-kube-api-access-rmxks\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.805777 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805577    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:12:18.805777 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805593    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:18.805777 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:18.805777 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.805998 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.805973    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 15 21:12:18.806300 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.806276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9rfg9\""
Sep 15 21:12:18.807732 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.807641    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 21:12:18.807905 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.807882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6n9xt\""
Sep 15 21:12:18.807999 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.807922    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 21:12:18.817981 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.817961    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"]
Sep 15 21:12:18.818868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.818847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx"]
Sep 15 21:12:18.906634 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a17472-2d2b-45ac-98c1-cce356627d5c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.906772 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s79m9\" (UniqueName: \"kubernetes.io/projected/99a17472-2d2b-45ac-98c1-cce356627d5c-kube-api-access-s79m9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.906772 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw75w\" (UniqueName: \"kubernetes.io/projected/8db3c092-254e-401c-ab8e-fa78000a69ce-kube-api-access-pw75w\") pod \"network-check-source-6f8b768455-jdbvb\" (UID: \"8db3c092-254e-401c-ab8e-fa78000a69ce\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb"
Sep 15 21:12:18.906772 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxks\" (UniqueName: \"kubernetes.io/projected/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-kube-api-access-rmxks\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.906772 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zgq\" (UniqueName: \"kubernetes.io/projected/95d5ba47-d85a-4922-bf29-9e1eedab4cd7-kube-api-access-z4zgq\") pod \"volume-data-source-validator-86695b7cc7-brljx\" (UID: \"95d5ba47-d85a-4922-bf29-9e1eedab4cd7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx"
Sep 15 21:12:18.906772 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:18.906997 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:18.906997 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:18.906889    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:12:18.906997 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.906997 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:18.906950    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls podName:90788b65-78a3-47a7-b1e7-78c3c55c3033 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:19.406933451 +0000 UTC m=+258.006845554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8nvsq" (UID: "90788b65-78a3-47a7-b1e7-78c3c55c3033") : secret "samples-operator-tls" not found
Sep 15 21:12:18.906997 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.906984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/41a05419-d00c-4312-bf2b-32c368d48def-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:18.907239 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.907028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.907239 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.907072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a17472-2d2b-45ac-98c1-cce356627d5c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.907239 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.907097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpp8\" (UniqueName: \"kubernetes.io/projected/90788b65-78a3-47a7-b1e7-78c3c55c3033-kube-api-access-wlpp8\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:18.907239 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:18.907143    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:18.907239 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:18.907211    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls podName:2553f442-cb63-4da5-9a2c-eeb5ba3db7d0 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:19.407195985 +0000 UTC m=+258.007108072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5vx6t" (UID: "2553f442-cb63-4da5-9a2c-eeb5ba3db7d0") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:18.907586 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.907568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a17472-2d2b-45ac-98c1-cce356627d5c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.907684 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.907665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.909130 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.909112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a17472-2d2b-45ac-98c1-cce356627d5c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:18.919035 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.919013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxks\" (UniqueName: \"kubernetes.io/projected/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-kube-api-access-rmxks\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:18.919198 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.919175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpp8\" (UniqueName: \"kubernetes.io/projected/90788b65-78a3-47a7-b1e7-78c3c55c3033-kube-api-access-wlpp8\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:18.919680 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:18.919656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s79m9\" (UniqueName: \"kubernetes.io/projected/99a17472-2d2b-45ac-98c1-cce356627d5c-kube-api-access-s79m9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-kj6pj\" (UID: \"99a17472-2d2b-45ac-98c1-cce356627d5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:19.007948 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.007918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zgq\" (UniqueName: \"kubernetes.io/projected/95d5ba47-d85a-4922-bf29-9e1eedab4cd7-kube-api-access-z4zgq\") pod \"volume-data-source-validator-86695b7cc7-brljx\" (UID: \"95d5ba47-d85a-4922-bf29-9e1eedab4cd7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx"
Sep 15 21:12:19.007948 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.007950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:19.008173 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.007989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/41a05419-d00c-4312-bf2b-32c368d48def-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:19.008173 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.008037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw75w\" (UniqueName: \"kubernetes.io/projected/8db3c092-254e-401c-ab8e-fa78000a69ce-kube-api-access-pw75w\") pod \"network-check-source-6f8b768455-jdbvb\" (UID: \"8db3c092-254e-401c-ab8e-fa78000a69ce\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb"
Sep 15 21:12:19.008173 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:19.008081    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 21:12:19.008173 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:19.008148    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert podName:41a05419-d00c-4312-bf2b-32c368d48def nodeName:}" failed. No retries permitted until 2026-09-15 21:12:19.508129643 +0000 UTC m=+258.108041758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jq6hz" (UID: "41a05419-d00c-4312-bf2b-32c368d48def") : secret "networking-console-plugin-cert" not found
Sep 15 21:12:19.008647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.008631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/41a05419-d00c-4312-bf2b-32c368d48def-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:19.016374 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.016341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zgq\" (UniqueName: \"kubernetes.io/projected/95d5ba47-d85a-4922-bf29-9e1eedab4cd7-kube-api-access-z4zgq\") pod \"volume-data-source-validator-86695b7cc7-brljx\" (UID: \"95d5ba47-d85a-4922-bf29-9e1eedab4cd7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx"
Sep 15 21:12:19.016509 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.016343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw75w\" (UniqueName: \"kubernetes.io/projected/8db3c092-254e-401c-ab8e-fa78000a69ce-kube-api-access-pw75w\") pod \"network-check-source-6f8b768455-jdbvb\" (UID: \"8db3c092-254e-401c-ab8e-fa78000a69ce\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb"
Sep 15 21:12:19.019987 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.019961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"
Sep 15 21:12:19.104647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.104618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb"
Sep 15 21:12:19.112676 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.112659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx"
Sep 15 21:12:19.290202 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.290127    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj"]
Sep 15 21:12:19.293398 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:19.293373    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a17472_2d2b_45ac_98c1_cce356627d5c.slice/crio-340813bcf1f039cecfda49d9674e8b506184989dd7793c045b3fa02775ef5769 WatchSource:0}: Error finding container 340813bcf1f039cecfda49d9674e8b506184989dd7793c045b3fa02775ef5769: Status 404 returned error can't find the container with id 340813bcf1f039cecfda49d9674e8b506184989dd7793c045b3fa02775ef5769
Sep 15 21:12:19.371069 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.371039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb"]
Sep 15 21:12:19.374288 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:19.374261    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db3c092_254e_401c_ab8e_fa78000a69ce.slice/crio-78ee52f00686b5a49bb9250fd5182dd14173bb81775a171bc1e555935408a88b WatchSource:0}: Error finding container 78ee52f00686b5a49bb9250fd5182dd14173bb81775a171bc1e555935408a88b: Status 404 returned error can't find the container with id 78ee52f00686b5a49bb9250fd5182dd14173bb81775a171bc1e555935408a88b
Sep 15 21:12:19.400237 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:19.400213    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d5ba47_d85a_4922_bf29_9e1eedab4cd7.slice/crio-b11a8624f063baaa13782525c2fd51b4e11b7823d93a29962ba5d30c5dcd47bb WatchSource:0}: Error finding container b11a8624f063baaa13782525c2fd51b4e11b7823d93a29962ba5d30c5dcd47bb: Status 404 returned error can't find the container with id b11a8624f063baaa13782525c2fd51b4e11b7823d93a29962ba5d30c5dcd47bb
Sep 15 21:12:19.400354 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.400337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx"]
Sep 15 21:12:19.410664 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.410637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:19.410763 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.410678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:19.410822 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:19.410808    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:19.410822 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:19.410812    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:12:19.410902 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:19.410866    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls podName:2553f442-cb63-4da5-9a2c-eeb5ba3db7d0 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:20.410850478 +0000 UTC m=+259.010762566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5vx6t" (UID: "2553f442-cb63-4da5-9a2c-eeb5ba3db7d0") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:19.410902 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:19.410879    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls podName:90788b65-78a3-47a7-b1e7-78c3c55c3033 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:20.410873033 +0000 UTC m=+259.010785120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8nvsq" (UID: "90788b65-78a3-47a7-b1e7-78c3c55c3033") : secret "samples-operator-tls" not found
Sep 15 21:12:19.511562 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.511528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:19.511706 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:19.511665    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 21:12:19.511759 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:19.511726    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert podName:41a05419-d00c-4312-bf2b-32c368d48def nodeName:}" failed. No retries permitted until 2026-09-15 21:12:20.511710013 +0000 UTC m=+259.111622111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jq6hz" (UID: "41a05419-d00c-4312-bf2b-32c368d48def") : secret "networking-console-plugin-cert" not found
Sep 15 21:12:19.653241 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.653146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb" event={"ID":"8db3c092-254e-401c-ab8e-fa78000a69ce","Type":"ContainerStarted","Data":"e9f8b8290378ac04f26b8feba9bc7e026d19ff525189139086efd9a6da1d2329"}
Sep 15 21:12:19.653241 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.653208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb" event={"ID":"8db3c092-254e-401c-ab8e-fa78000a69ce","Type":"ContainerStarted","Data":"78ee52f00686b5a49bb9250fd5182dd14173bb81775a171bc1e555935408a88b"}
Sep 15 21:12:19.654428 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.654397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj" event={"ID":"99a17472-2d2b-45ac-98c1-cce356627d5c","Type":"ContainerStarted","Data":"340813bcf1f039cecfda49d9674e8b506184989dd7793c045b3fa02775ef5769"}
Sep 15 21:12:19.655653 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.655630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx" event={"ID":"95d5ba47-d85a-4922-bf29-9e1eedab4cd7","Type":"ContainerStarted","Data":"b11a8624f063baaa13782525c2fd51b4e11b7823d93a29962ba5d30c5dcd47bb"}
Sep 15 21:12:19.669629 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:19.669590    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jdbvb" podStartSLOduration=1.6695736060000002 podStartE2EDuration="1.669573606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:19.668878748 +0000 UTC m=+258.268790859" watchObservedRunningTime="2026-09-15 21:12:19.669573606 +0000 UTC m=+258.269485756"
Sep 15 21:12:20.419560 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:20.419476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:20.419560 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:20.419551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:20.420113 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:20.419625    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:12:20.420113 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:20.419706    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:20.420113 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:20.419714    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls podName:90788b65-78a3-47a7-b1e7-78c3c55c3033 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:22.419692243 +0000 UTC m=+261.019604352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8nvsq" (UID: "90788b65-78a3-47a7-b1e7-78c3c55c3033") : secret "samples-operator-tls" not found
Sep 15 21:12:20.420113 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:20.419767    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls podName:2553f442-cb63-4da5-9a2c-eeb5ba3db7d0 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:22.419750889 +0000 UTC m=+261.019662981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5vx6t" (UID: "2553f442-cb63-4da5-9a2c-eeb5ba3db7d0") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:20.520159 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:20.520122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:20.520343 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:20.520270    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 21:12:20.520395 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:20.520355    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert podName:41a05419-d00c-4312-bf2b-32c368d48def nodeName:}" failed. No retries permitted until 2026-09-15 21:12:22.520339154 +0000 UTC m=+261.120251242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jq6hz" (UID: "41a05419-d00c-4312-bf2b-32c368d48def") : secret "networking-console-plugin-cert" not found
Sep 15 21:12:20.661479 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:20.661427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx" event={"ID":"95d5ba47-d85a-4922-bf29-9e1eedab4cd7","Type":"ContainerStarted","Data":"d2b0531be15ee66fa1b1d30a90d8917ef42756bc9fbd19d9cc54d6f07cc5fa68"}
Sep 15 21:12:20.677267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:20.677168    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-brljx" podStartSLOduration=1.5503230879999998 podStartE2EDuration="2.677149055s" totalImagesPullingTime="1.126825967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:20.675678807 +0000 UTC m=+259.275590918" watchObservedRunningTime="2026-09-15 21:12:20.677149055 +0000 UTC m=+259.277061167"
Sep 15 21:12:21.666527 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:21.666421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj" event={"ID":"99a17472-2d2b-45ac-98c1-cce356627d5c","Type":"ContainerStarted","Data":"0f0e60ab2634acbee75d1676f53cedb5a4078241f9aacbfb3c0d0d88f64486c6"}
Sep 15 21:12:21.681761 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:21.681705    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj" podStartSLOduration=1.645290416 podStartE2EDuration="3.681683148s" totalImagesPullingTime="2.036392732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:21.68076707 +0000 UTC m=+260.280679182" watchObservedRunningTime="2026-09-15 21:12:21.681683148 +0000 UTC m=+260.281595263"
Sep 15 21:12:22.436875 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.436842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:22.437039 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.436929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:22.437039 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:22.437007    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:12:22.437039 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:22.437007    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:22.437174 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:22.437057    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls podName:90788b65-78a3-47a7-b1e7-78c3c55c3033 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:26.437044119 +0000 UTC m=+265.036956207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8nvsq" (UID: "90788b65-78a3-47a7-b1e7-78c3c55c3033") : secret "samples-operator-tls" not found
Sep 15 21:12:22.437174 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:22.437071    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls podName:2553f442-cb63-4da5-9a2c-eeb5ba3db7d0 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:26.437063541 +0000 UTC m=+265.036975628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5vx6t" (UID: "2553f442-cb63-4da5-9a2c-eeb5ba3db7d0") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:22.537419 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.537387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:22.537571 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:22.537551    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 21:12:22.537629 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:22.537623    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert podName:41a05419-d00c-4312-bf2b-32c368d48def nodeName:}" failed. No retries permitted until 2026-09-15 21:12:26.537606879 +0000 UTC m=+265.137518967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jq6hz" (UID: "41a05419-d00c-4312-bf2b-32c368d48def") : secret "networking-console-plugin-cert" not found
Sep 15 21:12:22.975126 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.975031    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22"]
Sep 15 21:12:22.979801 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.979777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22"
Sep 15 21:12:22.982693 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.982674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 21:12:22.982809 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.982705    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4wslb\""
Sep 15 21:12:22.983126 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.983113    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 21:12:22.987915 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:22.987894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22"]
Sep 15 21:12:23.041866 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.041834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k85q\" (UniqueName: \"kubernetes.io/projected/fc04d009-45a3-42b9-9ada-6e5e7458c78b-kube-api-access-6k85q\") pod \"migrator-8b4d697c5-brk22\" (UID: \"fc04d009-45a3-42b9-9ada-6e5e7458c78b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22"
Sep 15 21:12:23.142404 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.142364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k85q\" (UniqueName: \"kubernetes.io/projected/fc04d009-45a3-42b9-9ada-6e5e7458c78b-kube-api-access-6k85q\") pod \"migrator-8b4d697c5-brk22\" (UID: \"fc04d009-45a3-42b9-9ada-6e5e7458c78b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22"
Sep 15 21:12:23.150713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.150683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k85q\" (UniqueName: \"kubernetes.io/projected/fc04d009-45a3-42b9-9ada-6e5e7458c78b-kube-api-access-6k85q\") pod \"migrator-8b4d697c5-brk22\" (UID: \"fc04d009-45a3-42b9-9ada-6e5e7458c78b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22"
Sep 15 21:12:23.288713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.288624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22"
Sep 15 21:12:23.524177 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.524147    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9299p"]
Sep 15 21:12:23.527532 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.527515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.529431 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.529395    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 21:12:23.529580 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.529466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 21:12:23.529580 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.529494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8r4hc\""
Sep 15 21:12:23.529722 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.529709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 21:12:23.529990 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.529977    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 21:12:23.540278 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.540214    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9299p"]
Sep 15 21:12:23.562541 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.562517    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22"]
Sep 15 21:12:23.565392 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:23.565373    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc04d009_45a3_42b9_9ada_6e5e7458c78b.slice/crio-60b98ceef683ba3a03b3ab6cca2bfce394c78ffb469ecfa938ee802cf505c180 WatchSource:0}: Error finding container 60b98ceef683ba3a03b3ab6cca2bfce394c78ffb469ecfa938ee802cf505c180: Status 404 returned error can't find the container with id 60b98ceef683ba3a03b3ab6cca2bfce394c78ffb469ecfa938ee802cf505c180
Sep 15 21:12:23.647617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.647570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9kl9\" (UniqueName: \"kubernetes.io/projected/7304d05c-061a-4692-89de-6e0fbdad7008-kube-api-access-m9kl9\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.647762 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.647649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7304d05c-061a-4692-89de-6e0fbdad7008-data-volume\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.647762 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.647733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.647857 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.647771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7304d05c-061a-4692-89de-6e0fbdad7008-crio-socket\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.647857 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.647811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.673958 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.673928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22" event={"ID":"fc04d009-45a3-42b9-9ada-6e5e7458c78b","Type":"ContainerStarted","Data":"60b98ceef683ba3a03b3ab6cca2bfce394c78ffb469ecfa938ee802cf505c180"}
Sep 15 21:12:23.749057 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.749024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9kl9\" (UniqueName: \"kubernetes.io/projected/7304d05c-061a-4692-89de-6e0fbdad7008-kube-api-access-m9kl9\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.749225 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.749083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7304d05c-061a-4692-89de-6e0fbdad7008-data-volume\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.749225 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.749123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.749225 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.749156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7304d05c-061a-4692-89de-6e0fbdad7008-crio-socket\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.749225 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.749200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.749404 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:23.749263    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:23.749404 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:23.749282    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:23.749404 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.749316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7304d05c-061a-4692-89de-6e0fbdad7008-crio-socket\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.749404 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:23.749340    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:24.24931817 +0000 UTC m=+262.849230259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:23.749404 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:23.749371    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:24.249352223 +0000 UTC m=+262.849264315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:23.749607 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.749482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7304d05c-061a-4692-89de-6e0fbdad7008-data-volume\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:23.758830 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:23.758812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9kl9\" (UniqueName: \"kubernetes.io/projected/7304d05c-061a-4692-89de-6e0fbdad7008-kube-api-access-m9kl9\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:24.253619 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:24.253569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:24.254099 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:24.253674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:24.254099 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:24.253728    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:24.254099 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:24.253787    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:24.254099 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:24.253821    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:25.25380193 +0000 UTC m=+263.853714035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:24.254099 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:24.253841    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:25.253830892 +0000 UTC m=+263.853742988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:25.262953 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:25.262917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:25.263363 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:25.263037    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:25.263363 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:25.263048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:25.263363 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:25.263097    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:27.263080729 +0000 UTC m=+265.862992830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:25.263363 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:25.263148    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:25.263363 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:25.263189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:27.263178753 +0000 UTC m=+265.863090846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:25.682755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:25.682672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22" event={"ID":"fc04d009-45a3-42b9-9ada-6e5e7458c78b","Type":"ContainerStarted","Data":"03f44bf08a1678c3ea70e593afb2a31e74a31443970ed331f5e371e31e6fde0e"}
Sep 15 21:12:25.682755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:25.682719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22" event={"ID":"fc04d009-45a3-42b9-9ada-6e5e7458c78b","Type":"ContainerStarted","Data":"ed531250f3122fc72e66e2de75d1d6e11f12653a5c037763688387bb3ba710fa"}
Sep 15 21:12:25.699868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:25.699828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-brk22" podStartSLOduration=2.461517477 podStartE2EDuration="3.699814071s" totalImagesPullingTime="1.238296594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:25.698494203 +0000 UTC m=+264.298406312" watchObservedRunningTime="2026-09-15 21:12:25.699814071 +0000 UTC m=+264.299726210"
Sep 15 21:12:26.472979 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:26.472927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:26.473368 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:26.473015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:26.473368 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:26.473081    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:12:26.473368 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:26.473118    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:26.473368 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:26.473145    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls podName:90788b65-78a3-47a7-b1e7-78c3c55c3033 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:34.473131326 +0000 UTC m=+273.073043413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8nvsq" (UID: "90788b65-78a3-47a7-b1e7-78c3c55c3033") : secret "samples-operator-tls" not found
Sep 15 21:12:26.473368 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:26.473162    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls podName:2553f442-cb63-4da5-9a2c-eeb5ba3db7d0 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:34.473150998 +0000 UTC m=+273.073063086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5vx6t" (UID: "2553f442-cb63-4da5-9a2c-eeb5ba3db7d0") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:26.574020 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:26.573989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:26.574169 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:26.574135    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 21:12:26.574221 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:26.574206    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert podName:41a05419-d00c-4312-bf2b-32c368d48def nodeName:}" failed. No retries permitted until 2026-09-15 21:12:34.574191979 +0000 UTC m=+273.174104072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jq6hz" (UID: "41a05419-d00c-4312-bf2b-32c368d48def") : secret "networking-console-plugin-cert" not found
Sep 15 21:12:26.595873 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:26.595853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zngks_00f0d21f-f617-40af-998d-895e688399ce/dns-node-resolver/0.log"
Sep 15 21:12:27.280723 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:27.280680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:27.280903 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:27.280746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:27.280903 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:27.280842    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:27.280967 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:27.280907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:31.280892597 +0000 UTC m=+269.880804684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:27.280967 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:27.280845    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:27.281036 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:27.280979    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:31.280966327 +0000 UTC m=+269.880878428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:27.395441 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:27.395419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lmfnh_fd84df4b-1e3b-4784-a5b7-c2ea08ab8877/node-ca/0.log"
Sep 15 21:12:28.197662 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:28.197635    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kv4ww_83cfd1f3-a4dc-4534-9151-0640d5a67623/serve-healthcheck-canary/0.log"
Sep 15 21:12:28.623406 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:28.623377    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-brk22_fc04d009-45a3-42b9-9ada-6e5e7458c78b/migrator/0.log"
Sep 15 21:12:28.794386 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:28.794360    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-brk22_fc04d009-45a3-42b9-9ada-6e5e7458c78b/graceful-termination/0.log"
Sep 15 21:12:28.996038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:28.996009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-kj6pj_99a17472-2d2b-45ac-98c1-cce356627d5c/kube-storage-version-migrator-operator/0.log"
Sep 15 21:12:31.312662 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:31.312627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:31.313085 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:31.312676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:31.313085 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:31.312766    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:31.313085 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:31.312769    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:31.313085 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:31.312826    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:39.312812774 +0000 UTC m=+277.912724861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:31.313085 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:31.312837    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls podName:7304d05c-061a-4692-89de-6e0fbdad7008 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:39.312831694 +0000 UTC m=+277.912743781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9299p" (UID: "7304d05c-061a-4692-89de-6e0fbdad7008") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:34.542783 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:34.542744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:34.543161 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:34.542851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:34.543161 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:34.542900    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:34.543161 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:34.542980    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls podName:2553f442-cb63-4da5-9a2c-eeb5ba3db7d0 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:50.542961732 +0000 UTC m=+289.142873824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5vx6t" (UID: "2553f442-cb63-4da5-9a2c-eeb5ba3db7d0") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:34.545358 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:34.545337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/90788b65-78a3-47a7-b1e7-78c3c55c3033-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8nvsq\" (UID: \"90788b65-78a3-47a7-b1e7-78c3c55c3033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:34.626962 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:34.626931    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"
Sep 15 21:12:34.643427 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:34.643404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:34.643605 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:34.643587    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 21:12:34.643685 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:34.643675    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert podName:41a05419-d00c-4312-bf2b-32c368d48def nodeName:}" failed. No retries permitted until 2026-09-15 21:12:50.643654477 +0000 UTC m=+289.243566567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jq6hz" (UID: "41a05419-d00c-4312-bf2b-32c368d48def") : secret "networking-console-plugin-cert" not found
Sep 15 21:12:34.892110 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:34.892040    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq"]
Sep 15 21:12:35.713805 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:35.713778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq" event={"ID":"90788b65-78a3-47a7-b1e7-78c3c55c3033","Type":"ContainerStarted","Data":"5a80844abf43e6a27f7e5c9fb89f948274449b24cac487857b77a21dd7669f9b"}
Sep 15 21:12:37.720504 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:37.720473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq" event={"ID":"90788b65-78a3-47a7-b1e7-78c3c55c3033","Type":"ContainerStarted","Data":"f21f5d0100062e3c9ca30e435794367e10ce7a3cea6d3669caaa3a03a4bc087e"}
Sep 15 21:12:37.720948 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:37.720529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq" event={"ID":"90788b65-78a3-47a7-b1e7-78c3c55c3033","Type":"ContainerStarted","Data":"ad902beb5ba7a19dd8655dbe96def0a00e240cc5eb4b4d8b60e832da914eced1"}
Sep 15 21:12:37.739402 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:37.736574    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8nvsq" podStartSLOduration=17.998350311 podStartE2EDuration="19.736553878s" totalImagesPullingTime="1.738203567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:37.735831591 +0000 UTC m=+276.335743700" watchObservedRunningTime="2026-09-15 21:12:37.736553878 +0000 UTC m=+276.336465988"
Sep 15 21:12:39.383862 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:39.383808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:39.384249 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:39.383912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:39.384366 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:39.384345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7304d05c-061a-4692-89de-6e0fbdad7008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:39.386223 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:39.386207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7304d05c-061a-4692-89de-6e0fbdad7008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9299p\" (UID: \"7304d05c-061a-4692-89de-6e0fbdad7008\") " pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:39.436588 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:39.436557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9299p"
Sep 15 21:12:39.708030 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:39.707997    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9299p"]
Sep 15 21:12:39.712025 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:39.712002    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7304d05c_061a_4692_89de_6e0fbdad7008.slice/crio-993f537aa30897f04aa03ebd6e069f6c7e0591aec5dc924b352a8651c484fc80 WatchSource:0}: Error finding container 993f537aa30897f04aa03ebd6e069f6c7e0591aec5dc924b352a8651c484fc80: Status 404 returned error can't find the container with id 993f537aa30897f04aa03ebd6e069f6c7e0591aec5dc924b352a8651c484fc80
Sep 15 21:12:39.728215 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:39.728184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9299p" event={"ID":"7304d05c-061a-4692-89de-6e0fbdad7008","Type":"ContainerStarted","Data":"993f537aa30897f04aa03ebd6e069f6c7e0591aec5dc924b352a8651c484fc80"}
Sep 15 21:12:40.732958 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:40.732926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9299p" event={"ID":"7304d05c-061a-4692-89de-6e0fbdad7008","Type":"ContainerStarted","Data":"7e9b97ff33a8ce435391b8ce5208567415ddb0a226234383823b648c78bea46d"}
Sep 15 21:12:40.733333 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:40.732980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9299p" event={"ID":"7304d05c-061a-4692-89de-6e0fbdad7008","Type":"ContainerStarted","Data":"8566dedb26516f70a0dc5113e550505bf89ef71ea2bfc428f854edb00793b080"}
Sep 15 21:12:42.413889 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:42.413845    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" podUID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"
Sep 15 21:12:42.414281 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:42.413845    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dlqdm" podUID="1fc041fc-3382-4159-b3be-8145231efc81"
Sep 15 21:12:42.742195 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:42.742146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9299p" event={"ID":"7304d05c-061a-4692-89de-6e0fbdad7008","Type":"ContainerStarted","Data":"50617455a2e9032a6f056574f1f196c2e1f55d6b8ad5e49f85e72f41f865a0d1"}
Sep 15 21:12:42.742392 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:42.742209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:12:42.742392 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:42.742172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:12:42.761008 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:42.760962    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9299p" podStartSLOduration=17.774618426 podStartE2EDuration="19.760948055s" totalImagesPullingTime="1.986329629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:12:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:42.759534799 +0000 UTC m=+281.359446910" watchObservedRunningTime="2026-09-15 21:12:42.760948055 +0000 UTC m=+281.360860194"
Sep 15 21:12:45.734216 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:45.734184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:12:45.734824 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:45.734246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:12:45.736956 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:45.736925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"image-registry-75f6f99d9-4wvm2\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") " pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:12:45.737070 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:45.736925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fc041fc-3382-4159-b3be-8145231efc81-metrics-tls\") pod \"dns-default-dlqdm\" (UID: \"1fc041fc-3382-4159-b3be-8145231efc81\") " pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:12:45.745338 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:45.745321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p5jgl\""
Sep 15 21:12:45.745434 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:45.745343    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4hkpc\""
Sep 15 21:12:45.753081 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:45.753061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:12:45.753081 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:45.753079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:12:46.033843 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:46.033734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dlqdm"]
Sep 15 21:12:46.035102 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:46.035082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75f6f99d9-4wvm2"]
Sep 15 21:12:46.036599 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:46.036576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc041fc_3382_4159_b3be_8145231efc81.slice/crio-6da61c9fc2c4de278342d970b3fb6e867aa4f9776e5e67d16889bb7be20abb8d WatchSource:0}: Error finding container 6da61c9fc2c4de278342d970b3fb6e867aa4f9776e5e67d16889bb7be20abb8d: Status 404 returned error can't find the container with id 6da61c9fc2c4de278342d970b3fb6e867aa4f9776e5e67d16889bb7be20abb8d
Sep 15 21:12:46.038332 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:46.038316    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dc67e0_5c59_4f5b_8a6a_7aef0ab84bbf.slice/crio-56bec9bc1c173ca83e33f7650e183b5805fb8cd37edb2737187fbc56f8cc2d1f WatchSource:0}: Error finding container 56bec9bc1c173ca83e33f7650e183b5805fb8cd37edb2737187fbc56f8cc2d1f: Status 404 returned error can't find the container with id 56bec9bc1c173ca83e33f7650e183b5805fb8cd37edb2737187fbc56f8cc2d1f
Sep 15 21:12:46.755175 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:46.755135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dlqdm" event={"ID":"1fc041fc-3382-4159-b3be-8145231efc81","Type":"ContainerStarted","Data":"6da61c9fc2c4de278342d970b3fb6e867aa4f9776e5e67d16889bb7be20abb8d"}
Sep 15 21:12:46.756610 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:46.756587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" event={"ID":"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf","Type":"ContainerStarted","Data":"7f9273c33bbb4f3d47a5f534fbb09ec0b8d6b14906327ec3d28995fc6f85be75"}
Sep 15 21:12:46.756694 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:46.756631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" event={"ID":"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf","Type":"ContainerStarted","Data":"56bec9bc1c173ca83e33f7650e183b5805fb8cd37edb2737187fbc56f8cc2d1f"}
Sep 15 21:12:46.756730 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:46.756706    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:12:46.777326 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:46.777287    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" podStartSLOduration=284.777272552 podStartE2EDuration="4m44.777272552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:46.774967058 +0000 UTC m=+285.374879167" watchObservedRunningTime="2026-09-15 21:12:46.777272552 +0000 UTC m=+285.377184660"
Sep 15 21:12:48.766204 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:48.766172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dlqdm" event={"ID":"1fc041fc-3382-4159-b3be-8145231efc81","Type":"ContainerStarted","Data":"3187f19368eece3b70318f1d549dc68c4f0d414102961f8718ce48950fb0ae99"}
Sep 15 21:12:48.766610 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:48.766224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dlqdm" event={"ID":"1fc041fc-3382-4159-b3be-8145231efc81","Type":"ContainerStarted","Data":"46902a77cca2fceb5e14a414a2f50cfee92a8eb24ef8fa739e5aaec9b0a5d933"}
Sep 15 21:12:48.766610 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:48.766287    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:12:48.781094 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:48.781054    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dlqdm" podStartSLOduration=251.714588407 podStartE2EDuration="4m13.781042162s" totalImagesPullingTime="2.066453755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:48.780755188 +0000 UTC m=+287.380667297" watchObservedRunningTime="2026-09-15 21:12:48.781042162 +0000 UTC m=+287.380954270"
Sep 15 21:12:50.569175 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:50.569126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:50.571615 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:50.571597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2553f442-cb63-4da5-9a2c-eeb5ba3db7d0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5vx6t\" (UID: \"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:50.669955 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:50.669919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:50.672154 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:50.672135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41a05419-d00c-4312-bf2b-32c368d48def-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jq6hz\" (UID: \"41a05419-d00c-4312-bf2b-32c368d48def\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:50.814429 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:50.814400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"
Sep 15 21:12:50.918110 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:50.918079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"
Sep 15 21:12:51.083363 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:51.083299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t"]
Sep 15 21:12:51.086592 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:51.086567    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2553f442_cb63_4da5_9a2c_eeb5ba3db7d0.slice/crio-d07663f7881cd950958105022e7b27eee139e3ef0892f2bcd1bd7e61d6b51230 WatchSource:0}: Error finding container d07663f7881cd950958105022e7b27eee139e3ef0892f2bcd1bd7e61d6b51230: Status 404 returned error can't find the container with id d07663f7881cd950958105022e7b27eee139e3ef0892f2bcd1bd7e61d6b51230
Sep 15 21:12:51.185420 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:51.185390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz"]
Sep 15 21:12:51.188704 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:51.188682    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a05419_d00c_4312_bf2b_32c368d48def.slice/crio-afb0d4c8c7bcfc2530d6fec82310b1ea6813cc967bf96e67b260cbe05dac43db WatchSource:0}: Error finding container afb0d4c8c7bcfc2530d6fec82310b1ea6813cc967bf96e67b260cbe05dac43db: Status 404 returned error can't find the container with id afb0d4c8c7bcfc2530d6fec82310b1ea6813cc967bf96e67b260cbe05dac43db
Sep 15 21:12:51.777344 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:51.777311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz" event={"ID":"41a05419-d00c-4312-bf2b-32c368d48def","Type":"ContainerStarted","Data":"afb0d4c8c7bcfc2530d6fec82310b1ea6813cc967bf96e67b260cbe05dac43db"}
Sep 15 21:12:51.778531 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:51.778499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t" event={"ID":"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0","Type":"ContainerStarted","Data":"d07663f7881cd950958105022e7b27eee139e3ef0892f2bcd1bd7e61d6b51230"}
Sep 15 21:12:53.511704 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.511675    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"]
Sep 15 21:12:53.514902 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.514886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"
Sep 15 21:12:53.516842 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.516820    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 21:12:53.516975 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.516886    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k2g7x\""
Sep 15 21:12:53.522671 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.522650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"]
Sep 15 21:12:53.586669 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.586634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3a8dacef-1ae1-423c-95cf-148348c154e2-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dlcdn\" (UID: \"3a8dacef-1ae1-423c-95cf-148348c154e2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"
Sep 15 21:12:53.687462 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.687428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3a8dacef-1ae1-423c-95cf-148348c154e2-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dlcdn\" (UID: \"3a8dacef-1ae1-423c-95cf-148348c154e2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"
Sep 15 21:12:53.687601 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:53.687579    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 15 21:12:53.687683 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:53.687652    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8dacef-1ae1-423c-95cf-148348c154e2-tls-certificates podName:3a8dacef-1ae1-423c-95cf-148348c154e2 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:54.187632208 +0000 UTC m=+292.787544296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/3a8dacef-1ae1-423c-95cf-148348c154e2-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-dlcdn" (UID: "3a8dacef-1ae1-423c-95cf-148348c154e2") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 15 21:12:53.787980 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.787881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz" event={"ID":"41a05419-d00c-4312-bf2b-32c368d48def","Type":"ContainerStarted","Data":"67c57598859e6db1fe3ae69b928c48637fd9c31d8da5ad2497bbc3084b48c838"}
Sep 15 21:12:53.789350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.789327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t" event={"ID":"2553f442-cb63-4da5-9a2c-eeb5ba3db7d0","Type":"ContainerStarted","Data":"e6b375fb333a6184fe9ef1e80996f262cb8b894e9b22ffa255de00e74a625cd2"}
Sep 15 21:12:53.802699 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.802656    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jq6hz" podStartSLOduration=33.973009055 podStartE2EDuration="35.802641531s" totalImagesPullingTime="1.829632476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:53.801896248 +0000 UTC m=+292.401808358" watchObservedRunningTime="2026-09-15 21:12:53.802641531 +0000 UTC m=+292.402553672"
Sep 15 21:12:53.816656 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:53.816599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5vx6t" podStartSLOduration=33.880576461 podStartE2EDuration="35.816577878s" totalImagesPullingTime="1.936001417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:53.816237943 +0000 UTC m=+292.416150055" watchObservedRunningTime="2026-09-15 21:12:53.816577878 +0000 UTC m=+292.416489989"
Sep 15 21:12:54.191988 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:54.191910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3a8dacef-1ae1-423c-95cf-148348c154e2-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dlcdn\" (UID: \"3a8dacef-1ae1-423c-95cf-148348c154e2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"
Sep 15 21:12:54.194369 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:54.194350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3a8dacef-1ae1-423c-95cf-148348c154e2-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dlcdn\" (UID: \"3a8dacef-1ae1-423c-95cf-148348c154e2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"
Sep 15 21:12:54.424103 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:54.424057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"
Sep 15 21:12:54.692956 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:54.692928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"]
Sep 15 21:12:54.696005 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:54.695968    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8dacef_1ae1_423c_95cf_148348c154e2.slice/crio-b9901f212224c4a6afd60d4f8bfcb96750d1b231069b8cd35d42a336528e23ff WatchSource:0}: Error finding container b9901f212224c4a6afd60d4f8bfcb96750d1b231069b8cd35d42a336528e23ff: Status 404 returned error can't find the container with id b9901f212224c4a6afd60d4f8bfcb96750d1b231069b8cd35d42a336528e23ff
Sep 15 21:12:54.794436 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:54.794355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn" event={"ID":"3a8dacef-1ae1-423c-95cf-148348c154e2","Type":"ContainerStarted","Data":"b9901f212224c4a6afd60d4f8bfcb96750d1b231069b8cd35d42a336528e23ff"}
Sep 15 21:12:56.803860 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:56.803824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn" event={"ID":"3a8dacef-1ae1-423c-95cf-148348c154e2","Type":"ContainerStarted","Data":"3f38a21b927bd09c9c466e40a1508c265d88ef8151f4d5fa22899ec977b2e9ba"}
Sep 15 21:12:56.804276 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:56.804051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"
Sep 15 21:12:56.808608 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:56.808586    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn"
Sep 15 21:12:56.818506 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:56.818469    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dlcdn" podStartSLOduration=2.502281786 podStartE2EDuration="3.818436364s" totalImagesPullingTime="1.316154578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:56.817208682 +0000 UTC m=+295.417120792" watchObservedRunningTime="2026-09-15 21:12:56.818436364 +0000 UTC m=+295.418348510"
Sep 15 21:12:56.895034 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:12:56.894999    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jlctj" podUID="eb9743c9-fe1a-4518-83aa-d84dbabcac9f"
Sep 15 21:12:57.572417 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.572387    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"]
Sep 15 21:12:57.575618 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.575603    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.577677 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.577655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 21:12:57.578510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.578486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dddjv\""
Sep 15 21:12:57.578655 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.578636    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 21:12:57.578704 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.578670    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 21:12:57.585310 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.585290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"]
Sep 15 21:12:57.613636 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.613612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/46bc392e-0b09-4237-9d53-295ead8ef704-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.613745 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.613639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bc392e-0b09-4237-9d53-295ead8ef704-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.613745 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.613656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468bx\" (UniqueName: \"kubernetes.io/projected/46bc392e-0b09-4237-9d53-295ead8ef704-kube-api-access-468bx\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.613845 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.613745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bc392e-0b09-4237-9d53-295ead8ef704-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.714327 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.714295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/46bc392e-0b09-4237-9d53-295ead8ef704-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.714500 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.714339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bc392e-0b09-4237-9d53-295ead8ef704-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.714500 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.714362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468bx\" (UniqueName: \"kubernetes.io/projected/46bc392e-0b09-4237-9d53-295ead8ef704-kube-api-access-468bx\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.714500 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.714460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bc392e-0b09-4237-9d53-295ead8ef704-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.715103 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.715080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bc392e-0b09-4237-9d53-295ead8ef704-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.716825 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.716801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bc392e-0b09-4237-9d53-295ead8ef704-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.716899 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.716870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/46bc392e-0b09-4237-9d53-295ead8ef704-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.722872 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.722853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-468bx\" (UniqueName: \"kubernetes.io/projected/46bc392e-0b09-4237-9d53-295ead8ef704-kube-api-access-468bx\") pod \"prometheus-operator-68bcd7f987-n89nc\" (UID: \"46bc392e-0b09-4237-9d53-295ead8ef704\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:57.885770 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:57.885682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"
Sep 15 21:12:58.163997 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:58.163911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-n89nc"]
Sep 15 21:12:58.169440 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:12:58.169416    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bc392e_0b09_4237_9d53_295ead8ef704.slice/crio-127297b826615f72c9ddb435e1ae2cbf046f2ca21785640328e75fd9dd59cbef WatchSource:0}: Error finding container 127297b826615f72c9ddb435e1ae2cbf046f2ca21785640328e75fd9dd59cbef: Status 404 returned error can't find the container with id 127297b826615f72c9ddb435e1ae2cbf046f2ca21785640328e75fd9dd59cbef
Sep 15 21:12:58.771650 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:58.771620    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dlqdm"
Sep 15 21:12:58.812535 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:58.812490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc" event={"ID":"46bc392e-0b09-4237-9d53-295ead8ef704","Type":"ContainerStarted","Data":"127297b826615f72c9ddb435e1ae2cbf046f2ca21785640328e75fd9dd59cbef"}
Sep 15 21:12:59.817662 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:59.817631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc" event={"ID":"46bc392e-0b09-4237-9d53-295ead8ef704","Type":"ContainerStarted","Data":"6e7908c5c60c8be4d0f7576d0a8ed2138a44038dfb06d65a4beda24b3bc9bd98"}
Sep 15 21:12:59.818036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:59.817682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc" event={"ID":"46bc392e-0b09-4237-9d53-295ead8ef704","Type":"ContainerStarted","Data":"42839314d69ea57b2b8808c7d493ec2b96479a38042ec43d5a52b232d2b699d6"}
Sep 15 21:12:59.832346 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:12:59.832305    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n89nc" podStartSLOduration=1.639961816 podStartE2EDuration="2.832289794s" totalImagesPullingTime="1.192327978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:59.832143426 +0000 UTC m=+298.432055536" watchObservedRunningTime="2026-09-15 21:12:59.832289794 +0000 UTC m=+298.432201904"
Sep 15 21:13:01.845247 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:01.845220    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 21:13:01.975657 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:01.975626    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"]
Sep 15 21:13:01.983230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:01.979708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:01.983230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:01.981756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lkpt9\""
Sep 15 21:13:01.983230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:01.982246    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 21:13:01.983230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:01.982248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 21:13:01.994810 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:01.994789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"]
Sep 15 21:13:02.031935 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.031906    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dtgmx"]
Sep 15 21:13:02.035158 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.035141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.037706 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.037687    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lwmf4\""
Sep 15 21:13:02.037819 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.037804    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 21:13:02.038138 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.038124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 21:13:02.041591 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.041574    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 21:13:02.046590 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-textfile\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.046679 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66sch\" (UniqueName: \"kubernetes.io/projected/fa32c370-5087-4f62-9842-10a68b16b8e6-kube-api-access-66sch\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.046679 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcb21e97-b29e-4372-9bc0-90e968051901-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.046766 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-root\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.046766 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-tls\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.046766 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa32c370-5087-4f62-9842-10a68b16b8e6-metrics-client-ca\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.046873 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb21e97-b29e-4372-9bc0-90e968051901-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.046873 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-wtmp\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.046873 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.046873 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcb21e97-b29e-4372-9bc0-90e968051901-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.047001 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44c6z\" (UniqueName: \"kubernetes.io/projected/fcb21e97-b29e-4372-9bc0-90e968051901-kube-api-access-44c6z\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.047001 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.046990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.047075 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.047011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-sys\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147469 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcb21e97-b29e-4372-9bc0-90e968051901-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.147469 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44c6z\" (UniqueName: \"kubernetes.io/projected/fcb21e97-b29e-4372-9bc0-90e968051901-kube-api-access-44c6z\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.147677 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147677 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-sys\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147677 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-textfile\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147677 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66sch\" (UniqueName: \"kubernetes.io/projected/fa32c370-5087-4f62-9842-10a68b16b8e6-kube-api-access-66sch\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147677 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcb21e97-b29e-4372-9bc0-90e968051901-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.147677 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-sys\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-root\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-tls\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa32c370-5087-4f62-9842-10a68b16b8e6-metrics-client-ca\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-root\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb21e97-b29e-4372-9bc0-90e968051901-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.147984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-wtmp\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-textfile\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.147984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.147926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.148617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.148074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-wtmp\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.148617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.148394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcb21e97-b29e-4372-9bc0-90e968051901-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.148617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.148419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.148617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.148420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa32c370-5087-4f62-9842-10a68b16b8e6-metrics-client-ca\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.150091 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.150062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.150721 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.150693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcb21e97-b29e-4372-9bc0-90e968051901-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.150828 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.150774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa32c370-5087-4f62-9842-10a68b16b8e6-node-exporter-tls\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.150828 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.150804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb21e97-b29e-4372-9bc0-90e968051901-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.156108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.156087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44c6z\" (UniqueName: \"kubernetes.io/projected/fcb21e97-b29e-4372-9bc0-90e968051901-kube-api-access-44c6z\") pod \"openshift-state-metrics-67dcf8c979-p528w\" (UID: \"fcb21e97-b29e-4372-9bc0-90e968051901\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.167437 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.167416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66sch\" (UniqueName: \"kubernetes.io/projected/fa32c370-5087-4f62-9842-10a68b16b8e6-kube-api-access-66sch\") pod \"node-exporter-dtgmx\" (UID: \"fa32c370-5087-4f62-9842-10a68b16b8e6\") " pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.288871 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.288838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"
Sep 15 21:13:02.343945 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.343689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dtgmx"
Sep 15 21:13:02.357692 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:13:02.357659    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa32c370_5087_4f62_9842_10a68b16b8e6.slice/crio-d4b6a375d88d29312abe00702da943689deed1861de7b79ea61df5e657ed9853 WatchSource:0}: Error finding container d4b6a375d88d29312abe00702da943689deed1861de7b79ea61df5e657ed9853: Status 404 returned error can't find the container with id d4b6a375d88d29312abe00702da943689deed1861de7b79ea61df5e657ed9853
Sep 15 21:13:02.574771 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.574744    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w"]
Sep 15 21:13:02.577386 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:13:02.577354    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb21e97_b29e_4372_9bc0_90e968051901.slice/crio-027e4f7b9b3168f070c9bb0b924a266f24424d164a842a410047e0ba387ded36 WatchSource:0}: Error finding container 027e4f7b9b3168f070c9bb0b924a266f24424d164a842a410047e0ba387ded36: Status 404 returned error can't find the container with id 027e4f7b9b3168f070c9bb0b924a266f24424d164a842a410047e0ba387ded36
Sep 15 21:13:02.830115 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.830023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtgmx" event={"ID":"fa32c370-5087-4f62-9842-10a68b16b8e6","Type":"ContainerStarted","Data":"d4b6a375d88d29312abe00702da943689deed1861de7b79ea61df5e657ed9853"}
Sep 15 21:13:02.832258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.832234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w" event={"ID":"fcb21e97-b29e-4372-9bc0-90e968051901","Type":"ContainerStarted","Data":"ff8c8d3d22e2ecad5a83ae1f38620fc30149c7f9f90bc80d299b275b6616f935"}
Sep 15 21:13:02.832392 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.832282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w" event={"ID":"fcb21e97-b29e-4372-9bc0-90e968051901","Type":"ContainerStarted","Data":"1970a22711e3f60161e794af68a1ee04ee4d2d62cb9cce99ccec57c930c2a290"}
Sep 15 21:13:02.832392 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:02.832318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w" event={"ID":"fcb21e97-b29e-4372-9bc0-90e968051901","Type":"ContainerStarted","Data":"027e4f7b9b3168f070c9bb0b924a266f24424d164a842a410047e0ba387ded36"}
Sep 15 21:13:03.053602 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.053572    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:13:03.059410 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.059383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.062713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.062684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x47qw\""
Sep 15 21:13:03.063169 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.063146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 21:13:03.063305 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.063206    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 21:13:03.064029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.063426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 21:13:03.064029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.063784    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 21:13:03.064029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.063852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 21:13:03.064029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.063882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 21:13:03.064277 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.064070    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 21:13:03.064277 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.064269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 21:13:03.064369 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.064355    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 21:13:03.076540 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.076520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:13:03.155258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjd7q\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-kube-api-access-xjd7q\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-config-volume\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-web-config\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-config-out\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155893 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155893 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.155893 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.155679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256684 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256684 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256921 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256921 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjd7q\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-kube-api-access-xjd7q\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256921 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-config-volume\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256921 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256921 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256921 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.256921 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-web-config\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.257270 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-config-out\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.257270 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.256954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.257270 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.257061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.257270 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.257087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.257570 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.257522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.258649 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.258622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.258792 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.258771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.260880 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.260282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.260880 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.260332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.260880 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.260554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-config-out\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.261278 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.261225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.261434 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.261414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.261531 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.261480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.262192 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.262172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-config-volume\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.262253 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.262172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.262522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.262504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-web-config\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.265371 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.265349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjd7q\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-kube-api-access-xjd7q\") pod \"alertmanager-main-0\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.371893 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.371862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:03.666243 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.666148    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:13:03.840087 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.839090    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa32c370-5087-4f62-9842-10a68b16b8e6" containerID="199b19c915c4e1c549ce45e409dbb1fdad9554f0c897f70ce0ceb9fc94f92c68" exitCode=0
Sep 15 21:13:03.840087 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.839147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtgmx" event={"ID":"fa32c370-5087-4f62-9842-10a68b16b8e6","Type":"ContainerDied","Data":"199b19c915c4e1c549ce45e409dbb1fdad9554f0c897f70ce0ceb9fc94f92c68"}
Sep 15 21:13:03.851751 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:13:03.851722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054987e8_9d70_4887_b2fa_0cd460964fa1.slice/crio-a52beb769baaabc74d027462c53650513f22232d9d7b97ff646f3aaf75240538 WatchSource:0}: Error finding container a52beb769baaabc74d027462c53650513f22232d9d7b97ff646f3aaf75240538: Status 404 returned error can't find the container with id a52beb769baaabc74d027462c53650513f22232d9d7b97ff646f3aaf75240538
Sep 15 21:13:03.853661 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:03.853643    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:13:04.047994 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.047955    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8656b74474-bnljb"]
Sep 15 21:13:04.051698 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.051676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.053901 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.053875    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7pnw4\""
Sep 15 21:13:04.054236 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.054145    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 21:13:04.054236 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.054150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 21:13:04.054386 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.054369    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 21:13:04.054424 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.054394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-45o2eqhghld00\""
Sep 15 21:13:04.054660 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.054644    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 21:13:04.054757 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.054742    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 21:13:04.064350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.064330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.064494 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.064361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-metrics-client-ca\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.064494 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.064394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-tls\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.064494 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.064462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.064651 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.064548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-grpc-tls\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.064651 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.064607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.064651 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.064635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjw4\" (UniqueName: \"kubernetes.io/projected/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-kube-api-access-wvjw4\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.064779 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.064665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.066148 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.066131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8656b74474-bnljb"]
Sep 15 21:13:04.165288 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.165263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.165408 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.165296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-metrics-client-ca\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.165408 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.165324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-tls\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.165408 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.165349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.165589 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.165406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-grpc-tls\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.165589 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.165435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.165589 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.165566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjw4\" (UniqueName: \"kubernetes.io/projected/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-kube-api-access-wvjw4\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.165920 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.165843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.166119 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.166094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-metrics-client-ca\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.168335 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.168280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-grpc-tls\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.168412 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.168396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.168584 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.168569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-tls\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.168700 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.168680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.168741 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.168720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.168777 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.168737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.173213 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.173193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjw4\" (UniqueName: \"kubernetes.io/projected/ecddf7c2-12d0-4c33-9b08-6ed39bd26c80-kube-api-access-wvjw4\") pod \"thanos-querier-8656b74474-bnljb\" (UID: \"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80\") " pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.361374 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.361334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:04.642520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.642481    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8656b74474-bnljb"]
Sep 15 21:13:04.647625 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:13:04.647596    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecddf7c2_12d0_4c33_9b08_6ed39bd26c80.slice/crio-eed3db7029a608e5139e616ab7f01302fd5aa4a82d0eda1e7fae793fcbf62c7e WatchSource:0}: Error finding container eed3db7029a608e5139e616ab7f01302fd5aa4a82d0eda1e7fae793fcbf62c7e: Status 404 returned error can't find the container with id eed3db7029a608e5139e616ab7f01302fd5aa4a82d0eda1e7fae793fcbf62c7e
Sep 15 21:13:04.845299 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.845253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerStarted","Data":"a52beb769baaabc74d027462c53650513f22232d9d7b97ff646f3aaf75240538"}
Sep 15 21:13:04.847836 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.847798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtgmx" event={"ID":"fa32c370-5087-4f62-9842-10a68b16b8e6","Type":"ContainerStarted","Data":"66c6461e385419e0f5404b73fc2b6755474c51354bfc8212a9a5eac88c25ce29"}
Sep 15 21:13:04.847971 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.847858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtgmx" event={"ID":"fa32c370-5087-4f62-9842-10a68b16b8e6","Type":"ContainerStarted","Data":"ca0d8845b941b61b3a575ab04db193c988d2580a8e5ff0d3a49b6e7f270c8f1e"}
Sep 15 21:13:04.850271 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.850243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w" event={"ID":"fcb21e97-b29e-4372-9bc0-90e968051901","Type":"ContainerStarted","Data":"ed3879635326968abec17f3ab884e6a6be54bdf554f1c8c4b1c9cfdfbc66c70a"}
Sep 15 21:13:04.852423 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.852397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb" event={"ID":"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80","Type":"ContainerStarted","Data":"eed3db7029a608e5139e616ab7f01302fd5aa4a82d0eda1e7fae793fcbf62c7e"}
Sep 15 21:13:04.870809 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.870768    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dtgmx" podStartSLOduration=2.8412962349999997 podStartE2EDuration="3.870751445s" totalImagesPullingTime="938.479811ms" totalInitContainerRuntime="90.975399ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:04.869806212 +0000 UTC m=+303.469718323" watchObservedRunningTime="2026-09-15 21:13:04.870751445 +0000 UTC m=+303.470663556"
Sep 15 21:13:04.888990 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:04.888939    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p528w" podStartSLOduration=2.684379019 podStartE2EDuration="3.888920862s" totalImagesPullingTime="1.204541843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:04.887775992 +0000 UTC m=+303.487688104" watchObservedRunningTime="2026-09-15 21:13:04.888920862 +0000 UTC m=+303.488832974"
Sep 15 21:13:05.757420 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:05.757384    2563 patch_prober.go:28] interesting pod/image-registry-75f6f99d9-4wvm2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 21:13:05.757799 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:05.757433    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" podUID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:13:05.857685 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:05.857638    2563 generic.go:356] "Generic (PLEG): container finished" podID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerID="319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a" exitCode=0
Sep 15 21:13:05.857864 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:05.857735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerDied","Data":"319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a"}
Sep 15 21:13:06.408886 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.408849    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85cf98b868-7brsf"]
Sep 15 21:13:06.413262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.413240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.415501 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.415480    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 21:13:06.415975 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.415918    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 21:13:06.416101 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.415989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 21:13:06.416101 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.416045    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 21:13:06.416224 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.416140    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8dqv2\""
Sep 15 21:13:06.416282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.416263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6tbtec3nsn3i0\""
Sep 15 21:13:06.424418 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.424397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85cf98b868-7brsf"]
Sep 15 21:13:06.492006 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.491984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-client-ca-bundle\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.492117 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.492027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f280aa87-d313-4477-9a52-8635473c5956-audit-log\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.492173 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.492121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f280aa87-d313-4477-9a52-8635473c5956-metrics-server-audit-profiles\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.492173 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.492147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hgnf\" (UniqueName: \"kubernetes.io/projected/f280aa87-d313-4477-9a52-8635473c5956-kube-api-access-7hgnf\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.492256 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.492183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-secret-metrics-server-tls\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.492256 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.492199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-secret-metrics-server-client-certs\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.492256 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.492237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f280aa87-d313-4477-9a52-8635473c5956-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.593211 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.593181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f280aa87-d313-4477-9a52-8635473c5956-metrics-server-audit-profiles\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.593354 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.593217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hgnf\" (UniqueName: \"kubernetes.io/projected/f280aa87-d313-4477-9a52-8635473c5956-kube-api-access-7hgnf\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.593354 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.593248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-secret-metrics-server-tls\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.593354 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.593273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-secret-metrics-server-client-certs\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.593354 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.593322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f280aa87-d313-4477-9a52-8635473c5956-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.593578 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.593368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-client-ca-bundle\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.593578 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.593408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f280aa87-d313-4477-9a52-8635473c5956-audit-log\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.593915 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.593867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f280aa87-d313-4477-9a52-8635473c5956-audit-log\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.594170 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.594148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f280aa87-d313-4477-9a52-8635473c5956-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.594387 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.594340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f280aa87-d313-4477-9a52-8635473c5956-metrics-server-audit-profiles\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.597736 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.597714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-client-ca-bundle\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.598199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.598174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-secret-metrics-server-tls\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.598956 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.598928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f280aa87-d313-4477-9a52-8635473c5956-secret-metrics-server-client-certs\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.602187 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.602160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hgnf\" (UniqueName: \"kubernetes.io/projected/f280aa87-d313-4477-9a52-8635473c5956-kube-api-access-7hgnf\") pod \"metrics-server-85cf98b868-7brsf\" (UID: \"f280aa87-d313-4477-9a52-8635473c5956\") " pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.723310 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.723226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:06.868353 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.868305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb" event={"ID":"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80","Type":"ContainerStarted","Data":"5e64e4d3e3b65ead3f3a8c775197b42034d36f865c243abc0161d369ae02f501"}
Sep 15 21:13:06.869021 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.868396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb" event={"ID":"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80","Type":"ContainerStarted","Data":"1c54123a3aefa5b89afc11bb742969762bcd2f1841bab75b8aac912abb5e5967"}
Sep 15 21:13:06.869021 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:06.868439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb" event={"ID":"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80","Type":"ContainerStarted","Data":"25dcb62fc8a3dab364b6ffeb2446c7018630e9b8881c0d05a7b501cf36da4abf"}
Sep 15 21:13:07.001741 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.001710    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85cf98b868-7brsf"]
Sep 15 21:13:07.004542 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:13:07.004511    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf280aa87_d313_4477_9a52_8635473c5956.slice/crio-d228e2f5fcd81bfc518c62b87ad1f0e3e5b490d7b15e4c388a67b9e42520d7c4 WatchSource:0}: Error finding container d228e2f5fcd81bfc518c62b87ad1f0e3e5b490d7b15e4c388a67b9e42520d7c4: Status 404 returned error can't find the container with id d228e2f5fcd81bfc518c62b87ad1f0e3e5b490d7b15e4c388a67b9e42520d7c4
Sep 15 21:13:07.755726 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.755700    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75f6f99d9-4wvm2"]
Sep 15 21:13:07.761384 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.761327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:13:07.877711 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.877683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf" event={"ID":"f280aa87-d313-4477-9a52-8635473c5956","Type":"ContainerStarted","Data":"d228e2f5fcd81bfc518c62b87ad1f0e3e5b490d7b15e4c388a67b9e42520d7c4"}
Sep 15 21:13:07.882875 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.882834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerStarted","Data":"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a"}
Sep 15 21:13:07.882985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.882947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerStarted","Data":"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4"}
Sep 15 21:13:07.883071 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.883040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerStarted","Data":"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1"}
Sep 15 21:13:07.883145 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.883127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerStarted","Data":"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71"}
Sep 15 21:13:07.883259 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.883208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerStarted","Data":"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40"}
Sep 15 21:13:07.894487 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:07.894436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:13:08.236044 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.235996    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:13:08.240771 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.240742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.245496 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 21:13:08.245496 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245136    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 21:13:08.245496 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 21:13:08.245496 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245303    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 21:13:08.245496 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245351    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 21:13:08.245496 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 21:13:08.246296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245633    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 21:13:08.246296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245759    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 21:13:08.246296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245875    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qhkjn\""
Sep 15 21:13:08.246296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.245969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 21:13:08.246296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.246036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b75v6v8k1ir65\""
Sep 15 21:13:08.246296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.246176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 21:13:08.247696 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.247540    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 21:13:08.247696 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.247562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 21:13:08.247844 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.247696    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 21:13:08.251011 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.250985    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 21:13:08.256716 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.256556    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:13:08.311927 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.311891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.311927 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.311935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szxv\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-kube-api-access-8szxv\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312189 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.311962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312189 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312189 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312189 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312189 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312189 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312189 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-web-config\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-config\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.312625 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.312530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-config-out\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.414801 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.414732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.414801 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.414787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.414815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.414860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-web-config\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.414887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.414910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.414962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.414992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-config\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-config-out\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szxv\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-kube-api-access-8szxv\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.415707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.415614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.416569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.416198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.418504 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.418270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.419956 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.419620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.419956 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.419709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.420645 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.420295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.420645 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.420385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.420645 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.420524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.420878 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.420777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.420993 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.420969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.421224 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.421201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.421583 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.421560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.422921 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.422872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-config-out\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.423468 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.423408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.423546 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.423503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.423546 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.423501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-config\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.424156 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.424112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-web-config\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.425839 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.425817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.426321 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.426302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.427478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.427432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szxv\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-kube-api-access-8szxv\") pod \"prometheus-k8s-0\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.562209 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.562120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:08.892980 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.892946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb" event={"ID":"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80","Type":"ContainerStarted","Data":"cb75cbae0fa1f57726c5163e9ba6d555ddc1f136cdf54ad6c3a3a34fd79659f1"}
Sep 15 21:13:08.893435 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.893037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb" event={"ID":"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80","Type":"ContainerStarted","Data":"52a9e7e6bb6f0b63e0b165c98dddc6e7ead52c513204d268321f8acf6617b264"}
Sep 15 21:13:08.893435 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.893071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:08.893435 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.893096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb" event={"ID":"ecddf7c2-12d0-4c33-9b08-6ed39bd26c80","Type":"ContainerStarted","Data":"23a6520b67d8ac41ad17c17d5ed62efe25ff4fee8a3a714ff983a4bce6176008"}
Sep 15 21:13:08.897373 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.897346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerStarted","Data":"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b"}
Sep 15 21:13:08.918197 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.918146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb" podStartSLOduration=2.012495417 podStartE2EDuration="4.917929626s" totalImagesPullingTime="2.905434209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:08.917009549 +0000 UTC m=+307.516921659" watchObservedRunningTime="2026-09-15 21:13:08.917929626 +0000 UTC m=+307.517841747"
Sep 15 21:13:08.949461 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:08.949402    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.30074202 podStartE2EDuration="5.949383155s" totalImagesPullingTime="2.632402908s" totalInitContainerRuntime="16.238227ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:08.947207868 +0000 UTC m=+307.547119978" watchObservedRunningTime="2026-09-15 21:13:08.949383155 +0000 UTC m=+307.549295314"
Sep 15 21:13:09.140857 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:09.140768    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:13:09.143786 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:13:09.143751    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac6a095_e31a_44ba_922b_ac723037a708.slice/crio-8e82572553f33e0beb05dc8d4c5160097d05187b6dea81e094073757307eb1df WatchSource:0}: Error finding container 8e82572553f33e0beb05dc8d4c5160097d05187b6dea81e094073757307eb1df: Status 404 returned error can't find the container with id 8e82572553f33e0beb05dc8d4c5160097d05187b6dea81e094073757307eb1df
Sep 15 21:13:09.905091 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:09.905047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf" event={"ID":"f280aa87-d313-4477-9a52-8635473c5956","Type":"ContainerStarted","Data":"c5148b29184e81db5f17fa26cabf2ebeb00d30b6f329faaac3705f3e09f3b141"}
Sep 15 21:13:09.906655 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:09.906630    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ac6a095-e31a-44ba-922b-ac723037a708" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73" exitCode=0
Sep 15 21:13:09.906770 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:09.906709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerDied","Data":"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"}
Sep 15 21:13:09.906770 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:09.906765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerStarted","Data":"8e82572553f33e0beb05dc8d4c5160097d05187b6dea81e094073757307eb1df"}
Sep 15 21:13:09.936699 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:09.936661    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf" podStartSLOduration=2.033448837 podStartE2EDuration="3.936648996s" totalImagesPullingTime="1.903200159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:09.93538174 +0000 UTC m=+308.535293851" watchObservedRunningTime="2026-09-15 21:13:09.936648996 +0000 UTC m=+308.536561105"
Sep 15 21:13:13.927361 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:13.927328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerStarted","Data":"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"}
Sep 15 21:13:13.927750 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:13.927390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerStarted","Data":"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"}
Sep 15 21:13:13.927750 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:13.927420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerStarted","Data":"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"}
Sep 15 21:13:13.927750 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:13.927469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerStarted","Data":"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"}
Sep 15 21:13:13.927750 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:13.927495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerStarted","Data":"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"}
Sep 15 21:13:13.927750 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:13.927525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerStarted","Data":"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"}
Sep 15 21:13:13.955034 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:13.954975    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.713000116 podStartE2EDuration="5.954961735s" totalImagesPullingTime="3.222768229s" totalInitContainerRuntime="19.19339ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:13.953846449 +0000 UTC m=+312.553758549" watchObservedRunningTime="2026-09-15 21:13:13.954961735 +0000 UTC m=+312.554873859"
Sep 15 21:13:14.912707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:14.912681    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8656b74474-bnljb"
Sep 15 21:13:18.562495 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:18.562429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:26.723598 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:26.723551    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:26.723975 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:26.723623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:32.775563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:32.775498    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" podUID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" containerName="registry" containerID="cri-o://7f9273c33bbb4f3d47a5f534fbb09ec0b8d6b14906327ec3d28995fc6f85be75" gracePeriod=30
Sep 15 21:13:33.004786 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.004752    2563 generic.go:356] "Generic (PLEG): container finished" podID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" containerID="7f9273c33bbb4f3d47a5f534fbb09ec0b8d6b14906327ec3d28995fc6f85be75" exitCode=0
Sep 15 21:13:33.004970 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.004803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" event={"ID":"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf","Type":"ContainerDied","Data":"7f9273c33bbb4f3d47a5f534fbb09ec0b8d6b14906327ec3d28995fc6f85be75"}
Sep 15 21:13:33.043798 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.043774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:13:33.050621 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.050598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp9hs\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-kube-api-access-bp9hs\") pod \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") "
Sep 15 21:13:33.050726 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.050647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-certificates\") pod \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") "
Sep 15 21:13:33.050726 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.050669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-image-registry-private-configuration\") pod \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") "
Sep 15 21:13:33.050726 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.050687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-bound-sa-token\") pod \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") "
Sep 15 21:13:33.050891 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.050734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") pod \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") "
Sep 15 21:13:33.050891 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.050780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-trusted-ca\") pod \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") "
Sep 15 21:13:33.050891 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.050806    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-installation-pull-secrets\") pod \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") "
Sep 15 21:13:33.050891 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.050855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-ca-trust-extracted\") pod \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\" (UID: \"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf\") "
Sep 15 21:13:33.051132 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.051099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:33.051669 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.051605    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:33.053259 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.053229    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:33.053418 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.053382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-kube-api-access-bp9hs" (OuterVolumeSpecName: "kube-api-access-bp9hs") pod "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"). InnerVolumeSpecName "kube-api-access-bp9hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:33.053418 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.053387    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:33.053570 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.053543    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:33.053829 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.053807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:33.061122 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.061097    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" (UID: "f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:13:33.152298 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.152266    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp9hs\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-kube-api-access-bp9hs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:13:33.152298 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.152293    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-certificates\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:13:33.152298 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.152302    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-image-registry-private-configuration\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:13:33.152569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.152313    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-bound-sa-token\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:13:33.152569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.152321    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-registry-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:13:33.152569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.152330    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-trusted-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:13:33.152569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.152338    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-installation-pull-secrets\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:13:33.152569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:33.152346    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf-ca-trust-extracted\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:13:34.015063 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:34.015027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2" event={"ID":"f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf","Type":"ContainerDied","Data":"56bec9bc1c173ca83e33f7650e183b5805fb8cd37edb2737187fbc56f8cc2d1f"}
Sep 15 21:13:34.015610 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:34.015093    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6f99d9-4wvm2"
Sep 15 21:13:34.015610 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:34.015105    2563 scope.go:117] "RemoveContainer" containerID="7f9273c33bbb4f3d47a5f534fbb09ec0b8d6b14906327ec3d28995fc6f85be75"
Sep 15 21:13:34.032642 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:34.032619    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75f6f99d9-4wvm2"]
Sep 15 21:13:34.036430 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:34.036409    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75f6f99d9-4wvm2"]
Sep 15 21:13:35.900175 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:35.900140    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" path="/var/lib/kubelet/pods/f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf/volumes"
Sep 15 21:13:46.728584 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:46.728548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:46.732534 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:46.732512    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85cf98b868-7brsf"
Sep 15 21:13:49.053910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:49.053885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kv4ww_83cfd1f3-a4dc-4534-9151-0640d5a67623/serve-healthcheck-canary/0.log"
Sep 15 21:13:53.093710 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:53.093623    2563 generic.go:356] "Generic (PLEG): container finished" podID="99a17472-2d2b-45ac-98c1-cce356627d5c" containerID="0f0e60ab2634acbee75d1676f53cedb5a4078241f9aacbfb3c0d0d88f64486c6" exitCode=0
Sep 15 21:13:53.093710 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:53.093680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj" event={"ID":"99a17472-2d2b-45ac-98c1-cce356627d5c","Type":"ContainerDied","Data":"0f0e60ab2634acbee75d1676f53cedb5a4078241f9aacbfb3c0d0d88f64486c6"}
Sep 15 21:13:53.094262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:53.094083    2563 scope.go:117] "RemoveContainer" containerID="0f0e60ab2634acbee75d1676f53cedb5a4078241f9aacbfb3c0d0d88f64486c6"
Sep 15 21:13:54.099857 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:13:54.099820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-kj6pj" event={"ID":"99a17472-2d2b-45ac-98c1-cce356627d5c","Type":"ContainerStarted","Data":"9a80f34911aa3333d699e5b7cf77fb50462a70bbfa50dbb965ccb560adb8ddff"}
Sep 15 21:14:08.563056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:08.563018    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:08.582334 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:08.582303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:09.181989 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:09.181958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:15.806815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:15.806782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:14:15.809230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:15.809208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9743c9-fe1a-4518-83aa-d84dbabcac9f-metrics-certs\") pod \"network-metrics-daemon-jlctj\" (UID: \"eb9743c9-fe1a-4518-83aa-d84dbabcac9f\") " pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:14:15.997717 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:15.997690    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lj4g\""
Sep 15 21:14:16.006759 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:16.006740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jlctj"
Sep 15 21:14:16.278169 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:16.278132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jlctj"]
Sep 15 21:14:16.281021 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:14:16.280995    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9743c9_fe1a_4518_83aa_d84dbabcac9f.slice/crio-49b5d528ae6bce8a95b694911ed1ea09e475124884287ea4f645aa9d05e45b5a WatchSource:0}: Error finding container 49b5d528ae6bce8a95b694911ed1ea09e475124884287ea4f645aa9d05e45b5a: Status 404 returned error can't find the container with id 49b5d528ae6bce8a95b694911ed1ea09e475124884287ea4f645aa9d05e45b5a
Sep 15 21:14:17.201307 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:17.201262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jlctj" event={"ID":"eb9743c9-fe1a-4518-83aa-d84dbabcac9f","Type":"ContainerStarted","Data":"49b5d528ae6bce8a95b694911ed1ea09e475124884287ea4f645aa9d05e45b5a"}
Sep 15 21:14:18.206712 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:18.206669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jlctj" event={"ID":"eb9743c9-fe1a-4518-83aa-d84dbabcac9f","Type":"ContainerStarted","Data":"c4b2bb800e3c6d8c5b985cf9cd83a805e6b35ddba99c808143ff1b79469de819"}
Sep 15 21:14:18.207111 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:18.206732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jlctj" event={"ID":"eb9743c9-fe1a-4518-83aa-d84dbabcac9f","Type":"ContainerStarted","Data":"8ff8d9f3b7335dd22354dffc9c0267990d696d6ea09228555f5d0874eeced92e"}
Sep 15 21:14:18.224004 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:18.223954    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jlctj" podStartSLOduration=374.9298638 podStartE2EDuration="6m16.223936958s" totalImagesPullingTime="1.294073158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:18.222605814 +0000 UTC m=+376.822517925" watchObservedRunningTime="2026-09-15 21:14:18.223936958 +0000 UTC m=+376.823849067"
Sep 15 21:14:22.332790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:22.332758    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:14:22.333353 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:22.333085    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="alertmanager" containerID="cri-o://c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40" gracePeriod=120
Sep 15 21:14:22.333353 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:22.333137    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy-metric" containerID="cri-o://93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a" gracePeriod=120
Sep 15 21:14:22.333353 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:22.333210    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy-web" containerID="cri-o://a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1" gracePeriod=120
Sep 15 21:14:22.333353 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:22.333236    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="prom-label-proxy" containerID="cri-o://f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b" gracePeriod=120
Sep 15 21:14:22.333353 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:22.333210    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="config-reloader" containerID="cri-o://a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71" gracePeriod=120
Sep 15 21:14:22.333671 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:22.333392    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy" containerID="cri-o://e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4" gracePeriod=120
Sep 15 21:14:23.232081 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.232046    2563 generic.go:356] "Generic (PLEG): container finished" podID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerID="f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b" exitCode=0
Sep 15 21:14:23.232081 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.232071    2563 generic.go:356] "Generic (PLEG): container finished" podID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerID="e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4" exitCode=0
Sep 15 21:14:23.232081 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.232077    2563 generic.go:356] "Generic (PLEG): container finished" podID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerID="a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71" exitCode=0
Sep 15 21:14:23.232081 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.232084    2563 generic.go:356] "Generic (PLEG): container finished" podID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerID="c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40" exitCode=0
Sep 15 21:14:23.232343 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.232107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerDied","Data":"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b"}
Sep 15 21:14:23.232343 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.232166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerDied","Data":"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4"}
Sep 15 21:14:23.232343 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.232193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerDied","Data":"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71"}
Sep 15 21:14:23.232343 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.232216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerDied","Data":"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40"}
Sep 15 21:14:23.581290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.581269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:23.766650 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-main-db\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766650 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-config-volume\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766650 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-web-config\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766650 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766640    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-trusted-ca-bundle\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-config-out\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-main-tls\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-cluster-tls-config\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjd7q\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-kube-api-access-xjd7q\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-metrics-client-ca\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-tls-assets\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-web\") pod \"054987e8-9d70-4887-b2fa-0cd460964fa1\" (UID: \"054987e8-9d70-4887-b2fa-0cd460964fa1\") "
Sep 15 21:14:23.766985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.766963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:23.767525 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.767025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:23.767525 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.767205    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-main-db\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.767525 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.767227    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.769491 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.769433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:23.770038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.769743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:23.770038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.769844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-config-out" (OuterVolumeSpecName: "config-out") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:23.770038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.769967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-kube-api-access-xjd7q" (OuterVolumeSpecName: "kube-api-access-xjd7q") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "kube-api-access-xjd7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:23.770254 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.770092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:23.770254 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.770242    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:23.770482 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.770433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-config-volume" (OuterVolumeSpecName: "config-volume") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:23.770638 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.770617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:23.772149 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.772131    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:23.776104 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.776080    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:23.781384 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.781365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-web-config" (OuterVolumeSpecName: "web-config") pod "054987e8-9d70-4887-b2fa-0cd460964fa1" (UID: "054987e8-9d70-4887-b2fa-0cd460964fa1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:23.868066 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868042    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/054987e8-9d70-4887-b2fa-0cd460964fa1-metrics-client-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868066 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868063    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-tls-assets\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868073    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868081    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868089    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-web-config\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868096    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868104    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/054987e8-9d70-4887-b2fa-0cd460964fa1-config-out\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868112    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868120    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-secret-alertmanager-main-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868127    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/054987e8-9d70-4887-b2fa-0cd460964fa1-cluster-tls-config\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:23.868199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:23.868135    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjd7q\" (UniqueName: \"kubernetes.io/projected/054987e8-9d70-4887-b2fa-0cd460964fa1-kube-api-access-xjd7q\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:24.239603 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.239568    2563 generic.go:356] "Generic (PLEG): container finished" podID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerID="93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a" exitCode=0
Sep 15 21:14:24.239603 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.239600    2563 generic.go:356] "Generic (PLEG): container finished" podID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerID="a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1" exitCode=0
Sep 15 21:14:24.239808 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.239644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerDied","Data":"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a"}
Sep 15 21:14:24.239808 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.239661    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:24.239808 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.239712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerDied","Data":"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1"}
Sep 15 21:14:24.239808 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.239742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"054987e8-9d70-4887-b2fa-0cd460964fa1","Type":"ContainerDied","Data":"a52beb769baaabc74d027462c53650513f22232d9d7b97ff646f3aaf75240538"}
Sep 15 21:14:24.239808 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.239771    2563 scope.go:117] "RemoveContainer" containerID="f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b"
Sep 15 21:14:24.248434 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.248412    2563 scope.go:117] "RemoveContainer" containerID="93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a"
Sep 15 21:14:24.255828 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.255808    2563 scope.go:117] "RemoveContainer" containerID="e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4"
Sep 15 21:14:24.259608 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.259588    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:14:24.263959 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.263938    2563 scope.go:117] "RemoveContainer" containerID="a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1"
Sep 15 21:14:24.264352 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.264329    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:14:24.270961 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.270946    2563 scope.go:117] "RemoveContainer" containerID="a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71"
Sep 15 21:14:24.278007 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.277991    2563 scope.go:117] "RemoveContainer" containerID="c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40"
Sep 15 21:14:24.285125 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.285085    2563 scope.go:117] "RemoveContainer" containerID="319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a"
Sep 15 21:14:24.292163 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.292139    2563 scope.go:117] "RemoveContainer" containerID="f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b"
Sep 15 21:14:24.292463 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:24.292423    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b\": container with ID starting with f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b not found: ID does not exist" containerID="f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b"
Sep 15 21:14:24.292523 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.292467    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b"} err="failed to get container status \"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b\": rpc error: code = NotFound desc = could not find container \"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b\": container with ID starting with f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b not found: ID does not exist"
Sep 15 21:14:24.292523 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.292501    2563 scope.go:117] "RemoveContainer" containerID="93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a"
Sep 15 21:14:24.292750 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:24.292736    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a\": container with ID starting with 93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a not found: ID does not exist" containerID="93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a"
Sep 15 21:14:24.292793 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.292753    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a"} err="failed to get container status \"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a\": rpc error: code = NotFound desc = could not find container \"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a\": container with ID starting with 93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a not found: ID does not exist"
Sep 15 21:14:24.292793 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.292765    2563 scope.go:117] "RemoveContainer" containerID="e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4"
Sep 15 21:14:24.293008 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:24.292991    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4\": container with ID starting with e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4 not found: ID does not exist" containerID="e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4"
Sep 15 21:14:24.293068 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.293017    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4"} err="failed to get container status \"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4\": rpc error: code = NotFound desc = could not find container \"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4\": container with ID starting with e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4 not found: ID does not exist"
Sep 15 21:14:24.293068 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.293039    2563 scope.go:117] "RemoveContainer" containerID="a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1"
Sep 15 21:14:24.293279 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:24.293261    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1\": container with ID starting with a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1 not found: ID does not exist" containerID="a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1"
Sep 15 21:14:24.293318 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.293284    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1"} err="failed to get container status \"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1\": rpc error: code = NotFound desc = could not find container \"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1\": container with ID starting with a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1 not found: ID does not exist"
Sep 15 21:14:24.293318 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.293298    2563 scope.go:117] "RemoveContainer" containerID="a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71"
Sep 15 21:14:24.293554 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:24.293538    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71\": container with ID starting with a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71 not found: ID does not exist" containerID="a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71"
Sep 15 21:14:24.293623 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.293560    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71"} err="failed to get container status \"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71\": rpc error: code = NotFound desc = could not find container \"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71\": container with ID starting with a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71 not found: ID does not exist"
Sep 15 21:14:24.293623 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.293578    2563 scope.go:117] "RemoveContainer" containerID="c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40"
Sep 15 21:14:24.293817 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:24.293802    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40\": container with ID starting with c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40 not found: ID does not exist" containerID="c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40"
Sep 15 21:14:24.293855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.293819    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40"} err="failed to get container status \"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40\": rpc error: code = NotFound desc = could not find container \"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40\": container with ID starting with c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40 not found: ID does not exist"
Sep 15 21:14:24.293855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.293832    2563 scope.go:117] "RemoveContainer" containerID="319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a"
Sep 15 21:14:24.294064 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:24.294051    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a\": container with ID starting with 319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a not found: ID does not exist" containerID="319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a"
Sep 15 21:14:24.294124 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294068    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a"} err="failed to get container status \"319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a\": rpc error: code = NotFound desc = could not find container \"319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a\": container with ID starting with 319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a not found: ID does not exist"
Sep 15 21:14:24.294124 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294085    2563 scope.go:117] "RemoveContainer" containerID="f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b"
Sep 15 21:14:24.294286 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294264    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b"} err="failed to get container status \"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b\": rpc error: code = NotFound desc = could not find container \"f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b\": container with ID starting with f781ece0b6574b9382a7ae3900364a72e88e59c14e3d9727ac5b7136c299209b not found: ID does not exist"
Sep 15 21:14:24.294336 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294287    2563 scope.go:117] "RemoveContainer" containerID="93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a"
Sep 15 21:14:24.294528 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294513    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a"} err="failed to get container status \"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a\": rpc error: code = NotFound desc = could not find container \"93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a\": container with ID starting with 93ff3618c0f11988e44ccae0bee7b34b8ebfb63ff92ab3ed24898af829fc984a not found: ID does not exist"
Sep 15 21:14:24.294528 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294527    2563 scope.go:117] "RemoveContainer" containerID="e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4"
Sep 15 21:14:24.294766 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294748    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4"} err="failed to get container status \"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4\": rpc error: code = NotFound desc = could not find container \"e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4\": container with ID starting with e4410bed854ac60935b03c8d258875c438fe102b348befa2d4dedf06fc07aff4 not found: ID does not exist"
Sep 15 21:14:24.294808 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294768    2563 scope.go:117] "RemoveContainer" containerID="a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1"
Sep 15 21:14:24.294975 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294958    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1"} err="failed to get container status \"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1\": rpc error: code = NotFound desc = could not find container \"a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1\": container with ID starting with a99b6cabd383716c4a30bbf3a837003917d8a35a3033e59f220fb61fffd500f1 not found: ID does not exist"
Sep 15 21:14:24.295016 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.294975    2563 scope.go:117] "RemoveContainer" containerID="a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71"
Sep 15 21:14:24.295207 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.295187    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71"} err="failed to get container status \"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71\": rpc error: code = NotFound desc = could not find container \"a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71\": container with ID starting with a04719e7ba346b8083b2eb285eca03761f048b17ec8901c3e9c356457983ee71 not found: ID does not exist"
Sep 15 21:14:24.295251 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.295209    2563 scope.go:117] "RemoveContainer" containerID="c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40"
Sep 15 21:14:24.295411 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.295397    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40"} err="failed to get container status \"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40\": rpc error: code = NotFound desc = could not find container \"c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40\": container with ID starting with c447743080184bc7bf00d69b576c84b51c2a6a4c797e9649d4352a4d45a51f40 not found: ID does not exist"
Sep 15 21:14:24.295469 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.295411    2563 scope.go:117] "RemoveContainer" containerID="319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a"
Sep 15 21:14:24.295611 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:24.295593    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a"} err="failed to get container status \"319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a\": rpc error: code = NotFound desc = could not find container \"319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a\": container with ID starting with 319c9f0a587e4c6ed6ecb73df6cb3b8145667bebe1f84cbdf33a8be26929646a not found: ID does not exist"
Sep 15 21:14:25.899458 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:25.899420    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" path="/var/lib/kubelet/pods/054987e8-9d70-4887-b2fa-0cd460964fa1/volumes"
Sep 15 21:14:26.557609 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.557569    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:26.557985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.557950    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="prometheus" containerID="cri-o://eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e" gracePeriod=600
Sep 15 21:14:26.558122 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.557990    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy" containerID="cri-o://68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903" gracePeriod=600
Sep 15 21:14:26.558122 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.558012    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="config-reloader" containerID="cri-o://5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c" gracePeriod=600
Sep 15 21:14:26.558122 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.558040    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy-web" containerID="cri-o://2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13" gracePeriod=600
Sep 15 21:14:26.558286 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.558012    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="thanos-sidecar" containerID="cri-o://272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca" gracePeriod=600
Sep 15 21:14:26.558286 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.558196    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c" gracePeriod=600
Sep 15 21:14:26.813875 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.813817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:26.902728 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902699    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-kube-rbac-proxy\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-2\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-config-out\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902782    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-thanos-prometheus-http-client-file\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-kubelet-serving-ca-bundle\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-trusted-ca-bundle\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-metrics-client-certs\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-metrics-client-ca\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-tls-assets\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8szxv\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-kube-api-access-8szxv\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.902993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-serving-certs-ca-bundle\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-web-config\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-tls\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-1\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-config\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.903267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-0\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.904370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903184    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-grpc-tls\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.904370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903211    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-db\") pod \"8ac6a095-e31a-44ba-922b-ac723037a708\" (UID: \"8ac6a095-e31a-44ba-922b-ac723037a708\") "
Sep 15 21:14:26.904370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:26.904370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:26.904370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903854    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:26.904370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903876    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:26.904370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903891    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:26.904370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.903955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:26.905622 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.905371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:26.905622 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.905489    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:26.905622 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.905577    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:26.906156 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.906126    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:26.906667 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.906634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:26.906667 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.906648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-config-out" (OuterVolumeSpecName: "config-out") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:26.906979 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.906674    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:26.908284 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.908253    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:26.908664 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.908601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:26.908756 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.908689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:26.908756 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.908712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:26.908955 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.908923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-config" (OuterVolumeSpecName: "config") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:26.909132 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.909102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-kube-api-access-8szxv" (OuterVolumeSpecName: "kube-api-access-8szxv") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "kube-api-access-8szxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:26.909236 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.909214    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:26.909294 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.909259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:26.919511 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:26.919491    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-web-config" (OuterVolumeSpecName: "web-config") pod "8ac6a095-e31a-44ba-922b-ac723037a708" (UID: "8ac6a095-e31a-44ba-922b-ac723037a708"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:27.004609 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004582    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004609 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004605    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-config\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004616    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004626    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004634    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-grpc-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004642    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-prometheus-k8s-db\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004649    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-kube-rbac-proxy\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004657    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ac6a095-e31a-44ba-922b-ac723037a708-config-out\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004665    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004674    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-metrics-client-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004682    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-metrics-client-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004690    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004698    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-tls-assets\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004706    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8szxv\" (UniqueName: \"kubernetes.io/projected/8ac6a095-e31a-44ba-922b-ac723037a708-kube-api-access-8szxv\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004713    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac6a095-e31a-44ba-922b-ac723037a708-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004721    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-web-config\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.004753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.004729    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ac6a095-e31a-44ba-922b-ac723037a708-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:14:27.257309 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257269    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ac6a095-e31a-44ba-922b-ac723037a708" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c" exitCode=0
Sep 15 21:14:27.257309 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257302    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ac6a095-e31a-44ba-922b-ac723037a708" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903" exitCode=0
Sep 15 21:14:27.257309 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257312    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ac6a095-e31a-44ba-922b-ac723037a708" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13" exitCode=0
Sep 15 21:14:27.257309 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257319    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ac6a095-e31a-44ba-922b-ac723037a708" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca" exitCode=0
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257330    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ac6a095-e31a-44ba-922b-ac723037a708" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c" exitCode=0
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257337    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ac6a095-e31a-44ba-922b-ac723037a708" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e" exitCode=0
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257342    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerDied","Data":"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"}
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerDied","Data":"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"}
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerDied","Data":"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"}
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257476    2563 scope.go:117] "RemoveContainer" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerDied","Data":"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"}
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerDied","Data":"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"}
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerDied","Data":"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"}
Sep 15 21:14:27.257663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.257568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ac6a095-e31a-44ba-922b-ac723037a708","Type":"ContainerDied","Data":"8e82572553f33e0beb05dc8d4c5160097d05187b6dea81e094073757307eb1df"}
Sep 15 21:14:27.265851 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.265825    2563 scope.go:117] "RemoveContainer" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"
Sep 15 21:14:27.274009 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.273995    2563 scope.go:117] "RemoveContainer" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"
Sep 15 21:14:27.281179 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.281159    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:27.281897 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.281884    2563 scope.go:117] "RemoveContainer" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"
Sep 15 21:14:27.285353 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.285333    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:27.288932 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.288916    2563 scope.go:117] "RemoveContainer" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"
Sep 15 21:14:27.295607 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.295594    2563 scope.go:117] "RemoveContainer" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"
Sep 15 21:14:27.302889 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.302871    2563 scope.go:117] "RemoveContainer" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"
Sep 15 21:14:27.309633 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.309618    2563 scope.go:117] "RemoveContainer" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"
Sep 15 21:14:27.309909 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:27.309892    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": container with ID starting with 4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c not found: ID does not exist" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"
Sep 15 21:14:27.309951 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.309920    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"} err="failed to get container status \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": rpc error: code = NotFound desc = could not find container \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": container with ID starting with 4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c not found: ID does not exist"
Sep 15 21:14:27.309951 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.309937    2563 scope.go:117] "RemoveContainer" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"
Sep 15 21:14:27.310182 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:27.310164    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": container with ID starting with 68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903 not found: ID does not exist" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"
Sep 15 21:14:27.310222 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.310188    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"} err="failed to get container status \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": rpc error: code = NotFound desc = could not find container \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": container with ID starting with 68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903 not found: ID does not exist"
Sep 15 21:14:27.310222 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.310204    2563 scope.go:117] "RemoveContainer" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"
Sep 15 21:14:27.310456 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:27.310424    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": container with ID starting with 2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13 not found: ID does not exist" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"
Sep 15 21:14:27.310529 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.310466    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"} err="failed to get container status \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": rpc error: code = NotFound desc = could not find container \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": container with ID starting with 2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13 not found: ID does not exist"
Sep 15 21:14:27.310529 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.310488    2563 scope.go:117] "RemoveContainer" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"
Sep 15 21:14:27.310740 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:27.310722    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": container with ID starting with 272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca not found: ID does not exist" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"
Sep 15 21:14:27.310776 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.310743    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"} err="failed to get container status \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": rpc error: code = NotFound desc = could not find container \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": container with ID starting with 272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca not found: ID does not exist"
Sep 15 21:14:27.310776 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.310756    2563 scope.go:117] "RemoveContainer" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"
Sep 15 21:14:27.311093 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:27.311070    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": container with ID starting with 5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c not found: ID does not exist" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"
Sep 15 21:14:27.311182 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.311099    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"} err="failed to get container status \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": rpc error: code = NotFound desc = could not find container \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": container with ID starting with 5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c not found: ID does not exist"
Sep 15 21:14:27.311245 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.311149    2563 scope.go:117] "RemoveContainer" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"
Sep 15 21:14:27.311536 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:27.311506    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": container with ID starting with eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e not found: ID does not exist" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"
Sep 15 21:14:27.311623 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.311536    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"} err="failed to get container status \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": rpc error: code = NotFound desc = could not find container \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": container with ID starting with eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e not found: ID does not exist"
Sep 15 21:14:27.311623 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.311555    2563 scope.go:117] "RemoveContainer" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"
Sep 15 21:14:27.311888 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:14:27.311860    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": container with ID starting with c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73 not found: ID does not exist" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"
Sep 15 21:14:27.311965 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.311895    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"} err="failed to get container status \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": rpc error: code = NotFound desc = could not find container \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": container with ID starting with c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73 not found: ID does not exist"
Sep 15 21:14:27.311965 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.311910    2563 scope.go:117] "RemoveContainer" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"
Sep 15 21:14:27.312164 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.312138    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"} err="failed to get container status \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": rpc error: code = NotFound desc = could not find container \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": container with ID starting with 4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c not found: ID does not exist"
Sep 15 21:14:27.312231 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.312164    2563 scope.go:117] "RemoveContainer" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"
Sep 15 21:14:27.312391 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.312374    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"} err="failed to get container status \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": rpc error: code = NotFound desc = could not find container \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": container with ID starting with 68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903 not found: ID does not exist"
Sep 15 21:14:27.312480 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.312390    2563 scope.go:117] "RemoveContainer" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"
Sep 15 21:14:27.312650 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.312632    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"} err="failed to get container status \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": rpc error: code = NotFound desc = could not find container \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": container with ID starting with 2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13 not found: ID does not exist"
Sep 15 21:14:27.312717 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.312650    2563 scope.go:117] "RemoveContainer" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"
Sep 15 21:14:27.312879 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.312858    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"} err="failed to get container status \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": rpc error: code = NotFound desc = could not find container \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": container with ID starting with 272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca not found: ID does not exist"
Sep 15 21:14:27.312936 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.312879    2563 scope.go:117] "RemoveContainer" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"
Sep 15 21:14:27.313035 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313016    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:27.313128 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313110    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"} err="failed to get container status \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": rpc error: code = NotFound desc = could not find container \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": container with ID starting with 5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c not found: ID does not exist"
Sep 15 21:14:27.313188 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313128    2563 scope.go:117] "RemoveContainer" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"
Sep 15 21:14:27.313380 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313361    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"} err="failed to get container status \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": rpc error: code = NotFound desc = could not find container \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": container with ID starting with eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e not found: ID does not exist"
Sep 15 21:14:27.313430 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313385    2563 scope.go:117] "RemoveContainer" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"
Sep 15 21:14:27.313430 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313369    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="prom-label-proxy"
Sep 15 21:14:27.313430 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313427    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="prom-label-proxy"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313437    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="config-reloader"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313461    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="config-reloader"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313473    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy-web"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313481    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy-web"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313492    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy-metric"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313497    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy-metric"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="config-reloader"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="config-reloader"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313515    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313520    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" containerName="registry"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313539    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" containerName="registry"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313550    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="init-config-reloader"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="init-config-reloader"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313563    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="init-config-reloader"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313568    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="init-config-reloader"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313574    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313579    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313586    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy-web"
Sep 15 21:14:27.313587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313591    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy-web"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313598    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="alertmanager"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313603    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="alertmanager"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313611    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy-thanos"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313619    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy-thanos"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="prometheus"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313637    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="prometheus"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313643    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="thanos-sidecar"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313650    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="thanos-sidecar"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313671    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"} err="failed to get container status \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": rpc error: code = NotFound desc = could not find container \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": container with ID starting with c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73 not found: ID does not exist"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313686    2563 scope.go:117] "RemoveContainer" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313728    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313738    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy-web"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313744    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="config-reloader"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313750    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313756    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="alertmanager"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313762    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="prometheus"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313769    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="prom-label-proxy"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313777    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="thanos-sidecar"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313783    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy-thanos"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313787    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="config-reloader"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313792    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="054987e8-9d70-4887-b2fa-0cd460964fa1" containerName="kube-rbac-proxy-metric"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313796    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" containerName="kube-rbac-proxy-web"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313802    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3dc67e0-5c59-4f5b-8a6a-7aef0ab84bbf" containerName="registry"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313883    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"} err="failed to get container status \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": rpc error: code = NotFound desc = could not find container \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": container with ID starting with 4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c not found: ID does not exist"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.313900    2563 scope.go:117] "RemoveContainer" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314070    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"} err="failed to get container status \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": rpc error: code = NotFound desc = could not find container \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": container with ID starting with 68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903 not found: ID does not exist"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314085    2563 scope.go:117] "RemoveContainer" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"
Sep 15 21:14:27.314312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314257    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"} err="failed to get container status \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": rpc error: code = NotFound desc = could not find container \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": container with ID starting with 2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13 not found: ID does not exist"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314269    2563 scope.go:117] "RemoveContainer" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314440    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"} err="failed to get container status \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": rpc error: code = NotFound desc = could not find container \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": container with ID starting with 272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca not found: ID does not exist"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314469    2563 scope.go:117] "RemoveContainer" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314642    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"} err="failed to get container status \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": rpc error: code = NotFound desc = could not find container \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": container with ID starting with 5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c not found: ID does not exist"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314652    2563 scope.go:117] "RemoveContainer" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314858    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"} err="failed to get container status \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": rpc error: code = NotFound desc = could not find container \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": container with ID starting with eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e not found: ID does not exist"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.314869    2563 scope.go:117] "RemoveContainer" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.315063    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"} err="failed to get container status \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": rpc error: code = NotFound desc = could not find container \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": container with ID starting with c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73 not found: ID does not exist"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.315079    2563 scope.go:117] "RemoveContainer" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.315284    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"} err="failed to get container status \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": rpc error: code = NotFound desc = could not find container \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": container with ID starting with 4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c not found: ID does not exist"
Sep 15 21:14:27.315331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.315297    2563 scope.go:117] "RemoveContainer" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"
Sep 15 21:14:27.315775 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.315536    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"} err="failed to get container status \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": rpc error: code = NotFound desc = could not find container \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": container with ID starting with 68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903 not found: ID does not exist"
Sep 15 21:14:27.315775 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.315552    2563 scope.go:117] "RemoveContainer" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"
Sep 15 21:14:27.315775 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.315764    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"} err="failed to get container status \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": rpc error: code = NotFound desc = could not find container \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": container with ID starting with 2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13 not found: ID does not exist"
Sep 15 21:14:27.315907 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.315778    2563 scope.go:117] "RemoveContainer" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"
Sep 15 21:14:27.316050 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.316032    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"} err="failed to get container status \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": rpc error: code = NotFound desc = could not find container \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": container with ID starting with 272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca not found: ID does not exist"
Sep 15 21:14:27.316097 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.316051    2563 scope.go:117] "RemoveContainer" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"
Sep 15 21:14:27.316296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.316274    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"} err="failed to get container status \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": rpc error: code = NotFound desc = could not find container \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": container with ID starting with 5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c not found: ID does not exist"
Sep 15 21:14:27.316345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.316298    2563 scope.go:117] "RemoveContainer" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"
Sep 15 21:14:27.316572 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.316549    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"} err="failed to get container status \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": rpc error: code = NotFound desc = could not find container \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": container with ID starting with eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e not found: ID does not exist"
Sep 15 21:14:27.316653 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.316574    2563 scope.go:117] "RemoveContainer" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"
Sep 15 21:14:27.316810 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.316794    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"} err="failed to get container status \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": rpc error: code = NotFound desc = could not find container \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": container with ID starting with c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73 not found: ID does not exist"
Sep 15 21:14:27.316856 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.316812    2563 scope.go:117] "RemoveContainer" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"
Sep 15 21:14:27.317128 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.317106    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"} err="failed to get container status \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": rpc error: code = NotFound desc = could not find container \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": container with ID starting with 4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c not found: ID does not exist"
Sep 15 21:14:27.317198 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.317129    2563 scope.go:117] "RemoveContainer" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"
Sep 15 21:14:27.317392 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.317373    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"} err="failed to get container status \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": rpc error: code = NotFound desc = could not find container \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": container with ID starting with 68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903 not found: ID does not exist"
Sep 15 21:14:27.317483 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.317394    2563 scope.go:117] "RemoveContainer" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"
Sep 15 21:14:27.317729 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.317700    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"} err="failed to get container status \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": rpc error: code = NotFound desc = could not find container \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": container with ID starting with 2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13 not found: ID does not exist"
Sep 15 21:14:27.317729 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.317728    2563 scope.go:117] "RemoveContainer" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"
Sep 15 21:14:27.317950 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.317933    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"} err="failed to get container status \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": rpc error: code = NotFound desc = could not find container \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": container with ID starting with 272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca not found: ID does not exist"
Sep 15 21:14:27.318005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.317952    2563 scope.go:117] "RemoveContainer" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"
Sep 15 21:14:27.318187 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.318170    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"} err="failed to get container status \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": rpc error: code = NotFound desc = could not find container \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": container with ID starting with 5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c not found: ID does not exist"
Sep 15 21:14:27.318246 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.318187    2563 scope.go:117] "RemoveContainer" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"
Sep 15 21:14:27.318419 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.318399    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"} err="failed to get container status \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": rpc error: code = NotFound desc = could not find container \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": container with ID starting with eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e not found: ID does not exist"
Sep 15 21:14:27.318508 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.318420    2563 scope.go:117] "RemoveContainer" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"
Sep 15 21:14:27.318725 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.318707    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"} err="failed to get container status \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": rpc error: code = NotFound desc = could not find container \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": container with ID starting with c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73 not found: ID does not exist"
Sep 15 21:14:27.318774 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.318725    2563 scope.go:117] "RemoveContainer" containerID="4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"
Sep 15 21:14:27.318996 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.318976    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c"} err="failed to get container status \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": rpc error: code = NotFound desc = could not find container \"4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c\": container with ID starting with 4c4dcf691781a766207dee8bd7e03024abc7ccc522ec2cac176a2cd16ffdd65c not found: ID does not exist"
Sep 15 21:14:27.319048 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.318996    2563 scope.go:117] "RemoveContainer" containerID="68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"
Sep 15 21:14:27.319244 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.319226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.319340 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.319242    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903"} err="failed to get container status \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": rpc error: code = NotFound desc = could not find container \"68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903\": container with ID starting with 68e3de1d598c044d8d217ba6f0ddcb3693ec5d069d7993f878b5c9fbe6d27903 not found: ID does not exist"
Sep 15 21:14:27.319340 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.319259    2563 scope.go:117] "RemoveContainer" containerID="2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"
Sep 15 21:14:27.319651 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.319554    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13"} err="failed to get container status \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": rpc error: code = NotFound desc = could not find container \"2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13\": container with ID starting with 2255827e360c4dd7d15d8c1be64adb66ffb61aeec8a64cda7595af8b85e32a13 not found: ID does not exist"
Sep 15 21:14:27.319651 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.319580    2563 scope.go:117] "RemoveContainer" containerID="272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"
Sep 15 21:14:27.319859 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.319830    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca"} err="failed to get container status \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": rpc error: code = NotFound desc = could not find container \"272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca\": container with ID starting with 272e36922e6c7510eefbd86b156cbeebeba1046300f71c2df0578dee4a3543ca not found: ID does not exist"
Sep 15 21:14:27.319964 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.319861    2563 scope.go:117] "RemoveContainer" containerID="5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"
Sep 15 21:14:27.320119 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.320103    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c"} err="failed to get container status \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": rpc error: code = NotFound desc = could not find container \"5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c\": container with ID starting with 5d1bcff6ff7c553482f5fe5ea48e5b78543e3aa61e85e5cb320bbba6fa0a6b5c not found: ID does not exist"
Sep 15 21:14:27.320172 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.320120    2563 scope.go:117] "RemoveContainer" containerID="eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"
Sep 15 21:14:27.320358 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.320340    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e"} err="failed to get container status \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": rpc error: code = NotFound desc = could not find container \"eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e\": container with ID starting with eb53372a2df0fd23357b3a9b8a2a994d620fa6e9b707dd30f98002fb603b620e not found: ID does not exist"
Sep 15 21:14:27.320430 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.320360    2563 scope.go:117] "RemoveContainer" containerID="c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"
Sep 15 21:14:27.320637 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.320620    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73"} err="failed to get container status \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": rpc error: code = NotFound desc = could not find container \"c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73\": container with ID starting with c2730325ed328bec9341bd7679862067234d77e7233bb377a693fc69ac55ff73 not found: ID does not exist"
Sep 15 21:14:27.321595 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b75v6v8k1ir65\""
Sep 15 21:14:27.321595 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321478    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 21:14:27.321595 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 21:14:27.321595 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321622    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 21:14:27.321855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321653    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 21:14:27.321855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 21:14:27.321855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 21:14:27.321855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321691    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 21:14:27.321855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 21:14:27.321855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.321783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 21:14:27.322296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.322272    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 21:14:27.322397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.322339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 21:14:27.322397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.322350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qhkjn\""
Sep 15 21:14:27.322549 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.322502    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 21:14:27.325260 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.325240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 21:14:27.327998 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.327977    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 21:14:27.336136 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.336117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:27.407783 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.407783 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.407980 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.407980 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.407980 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.407980 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.407980 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.407980 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.407987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-web-config\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-config\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5rc\" (UniqueName: \"kubernetes.io/projected/feaf4686-0714-4b1c-934f-569598a45200-kube-api-access-jb5rc\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/feaf4686-0714-4b1c-934f-569598a45200-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.408397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.408299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/feaf4686-0714-4b1c-934f-569598a45200-config-out\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509228 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509228 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/feaf4686-0714-4b1c-934f-569598a45200-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509228 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/feaf4686-0714-4b1c-934f-569598a45200-config-out\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-web-config\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-config\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.509963 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5rc\" (UniqueName: \"kubernetes.io/projected/feaf4686-0714-4b1c-934f-569598a45200-kube-api-access-jb5rc\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.510390 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.509996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.510390 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.510017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.510390 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.510368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.512499 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.510815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.512499 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.510957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.512499 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.510979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.512499 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.511520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.513799 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.513061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/feaf4686-0714-4b1c-934f-569598a45200-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.513799 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.513136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/feaf4686-0714-4b1c-934f-569598a45200-config-out\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.513799 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.513254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.513799 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.513752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.514171 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.514146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/feaf4686-0714-4b1c-934f-569598a45200-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.514257 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.514233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.514502 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.514479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-config\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.514747 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.514729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-web-config\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.515047 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.515021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.515306 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.515290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.515699 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.515676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.516245 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.516226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feaf4686-0714-4b1c-934f-569598a45200-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.517743 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.517725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5rc\" (UniqueName: \"kubernetes.io/projected/feaf4686-0714-4b1c-934f-569598a45200-kube-api-access-jb5rc\") pod \"prometheus-k8s-0\" (UID: \"feaf4686-0714-4b1c-934f-569598a45200\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.629005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.628976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:27.903971 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.903893    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac6a095-e31a-44ba-922b-ac723037a708" path="/var/lib/kubelet/pods/8ac6a095-e31a-44ba-922b-ac723037a708/volumes"
Sep 15 21:14:27.914438 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:27.914417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:27.916463 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:14:27.916428    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaf4686_0714_4b1c_934f_569598a45200.slice/crio-61ed7bfbc377db59032e1eb08fa0735670537c4c7c58f4d86b375caf04dd5a4e WatchSource:0}: Error finding container 61ed7bfbc377db59032e1eb08fa0735670537c4c7c58f4d86b375caf04dd5a4e: Status 404 returned error can't find the container with id 61ed7bfbc377db59032e1eb08fa0735670537c4c7c58f4d86b375caf04dd5a4e
Sep 15 21:14:28.264794 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:28.264761    2563 generic.go:356] "Generic (PLEG): container finished" podID="feaf4686-0714-4b1c-934f-569598a45200" containerID="80e71f957483d7deae142b6f10da0e5fbe4c4f071b91992a0b4ddc9c207d447a" exitCode=0
Sep 15 21:14:28.264794 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:28.264800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feaf4686-0714-4b1c-934f-569598a45200","Type":"ContainerDied","Data":"80e71f957483d7deae142b6f10da0e5fbe4c4f071b91992a0b4ddc9c207d447a"}
Sep 15 21:14:28.264988 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:28.264847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feaf4686-0714-4b1c-934f-569598a45200","Type":"ContainerStarted","Data":"61ed7bfbc377db59032e1eb08fa0735670537c4c7c58f4d86b375caf04dd5a4e"}
Sep 15 21:14:29.272896 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:29.272862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feaf4686-0714-4b1c-934f-569598a45200","Type":"ContainerStarted","Data":"50032fe40091bb1511f8f6dbc916ae8c1319312463d40791e20602a71f396316"}
Sep 15 21:14:29.273262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:29.272926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feaf4686-0714-4b1c-934f-569598a45200","Type":"ContainerStarted","Data":"1c1d64334bd760330d03ac8be446f5e06f29cc178377d122a335c88e2af1bba3"}
Sep 15 21:14:29.273262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:29.272954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feaf4686-0714-4b1c-934f-569598a45200","Type":"ContainerStarted","Data":"a0253261ba2594ecf2937a500b8659bfacb75c4462322e0d7a588097f045248f"}
Sep 15 21:14:29.273262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:29.272980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feaf4686-0714-4b1c-934f-569598a45200","Type":"ContainerStarted","Data":"9cc95d14d4ab225cba7590864a97c69f2fb817358c5e66708948e75880b0b184"}
Sep 15 21:14:29.273262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:29.273003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feaf4686-0714-4b1c-934f-569598a45200","Type":"ContainerStarted","Data":"f605b75a1ac8079782b5ee1f9e88df24185efc647e5b73ecd610f3cef8c32c00"}
Sep 15 21:14:29.273262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:29.273029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feaf4686-0714-4b1c-934f-569598a45200","Type":"ContainerStarted","Data":"29519d6dfbb01c1fbddf3fedcfc9370c7e6b55946649da2c8ca39fc145ac35e4"}
Sep 15 21:14:29.303792 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:29.303729    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.282431215 podStartE2EDuration="2.303707925s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.27671ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:29.30088369 +0000 UTC m=+387.900795800" watchObservedRunningTime="2026-09-15 21:14:29.303707925 +0000 UTC m=+387.903620039"
Sep 15 21:14:32.629804 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:14:32.629761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:15:27.629914 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:27.629813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:15:27.645882 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:27.645855    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:15:28.525019 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:28.524992    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:15:57.140137 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.140102    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k2mvg"]
Sep 15 21:15:57.143308 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.143292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k2mvg"
Sep 15 21:15:57.146049 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.146025    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 21:15:57.146155 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.146032    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7dt24\""
Sep 15 21:15:57.146360 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.146346    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 21:15:57.154875 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.154851    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k2mvg"]
Sep 15 21:15:57.219483 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.219460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmzr\" (UniqueName: \"kubernetes.io/projected/3a75f8b0-5cf9-420a-8b32-650c6f389c9e-kube-api-access-6hmzr\") pod \"cert-manager-55b4d954d5-k2mvg\" (UID: \"3a75f8b0-5cf9-420a-8b32-650c6f389c9e\") " pod="cert-manager/cert-manager-55b4d954d5-k2mvg"
Sep 15 21:15:57.219588 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.219510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a75f8b0-5cf9-420a-8b32-650c6f389c9e-bound-sa-token\") pod \"cert-manager-55b4d954d5-k2mvg\" (UID: \"3a75f8b0-5cf9-420a-8b32-650c6f389c9e\") " pod="cert-manager/cert-manager-55b4d954d5-k2mvg"
Sep 15 21:15:57.319904 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.319872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a75f8b0-5cf9-420a-8b32-650c6f389c9e-bound-sa-token\") pod \"cert-manager-55b4d954d5-k2mvg\" (UID: \"3a75f8b0-5cf9-420a-8b32-650c6f389c9e\") " pod="cert-manager/cert-manager-55b4d954d5-k2mvg"
Sep 15 21:15:57.320046 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.319927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmzr\" (UniqueName: \"kubernetes.io/projected/3a75f8b0-5cf9-420a-8b32-650c6f389c9e-kube-api-access-6hmzr\") pod \"cert-manager-55b4d954d5-k2mvg\" (UID: \"3a75f8b0-5cf9-420a-8b32-650c6f389c9e\") " pod="cert-manager/cert-manager-55b4d954d5-k2mvg"
Sep 15 21:15:57.327588 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.327558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a75f8b0-5cf9-420a-8b32-650c6f389c9e-bound-sa-token\") pod \"cert-manager-55b4d954d5-k2mvg\" (UID: \"3a75f8b0-5cf9-420a-8b32-650c6f389c9e\") " pod="cert-manager/cert-manager-55b4d954d5-k2mvg"
Sep 15 21:15:57.327699 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.327652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmzr\" (UniqueName: \"kubernetes.io/projected/3a75f8b0-5cf9-420a-8b32-650c6f389c9e-kube-api-access-6hmzr\") pod \"cert-manager-55b4d954d5-k2mvg\" (UID: \"3a75f8b0-5cf9-420a-8b32-650c6f389c9e\") " pod="cert-manager/cert-manager-55b4d954d5-k2mvg"
Sep 15 21:15:57.467266 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.467239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k2mvg"
Sep 15 21:15:57.737970 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:57.737893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k2mvg"]
Sep 15 21:15:57.741491 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:15:57.741439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a75f8b0_5cf9_420a_8b32_650c6f389c9e.slice/crio-b1d7ceb695dff3dee7ddaa1c72b0741c5aec2ee16968564a4c2cc56b429c9b15 WatchSource:0}: Error finding container b1d7ceb695dff3dee7ddaa1c72b0741c5aec2ee16968564a4c2cc56b429c9b15: Status 404 returned error can't find the container with id b1d7ceb695dff3dee7ddaa1c72b0741c5aec2ee16968564a4c2cc56b429c9b15
Sep 15 21:15:58.627900 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:15:58.627863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k2mvg" event={"ID":"3a75f8b0-5cf9-420a-8b32-650c6f389c9e","Type":"ContainerStarted","Data":"b1d7ceb695dff3dee7ddaa1c72b0741c5aec2ee16968564a4c2cc56b429c9b15"}
Sep 15 21:16:01.642485 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:01.642434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k2mvg" event={"ID":"3a75f8b0-5cf9-420a-8b32-650c6f389c9e","Type":"ContainerStarted","Data":"916eb09d66e9abc378b05026e6c50f29e71bfbe4184e50f24638f089e6511829"}
Sep 15 21:16:01.657271 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:01.657218    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-k2mvg" podStartSLOduration=1.6293956710000002 podStartE2EDuration="4.657188931s" totalImagesPullingTime="3.02779326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:01.657084481 +0000 UTC m=+480.256996590" watchObservedRunningTime="2026-09-15 21:16:01.657188931 +0000 UTC m=+480.257101042"
Sep 15 21:16:05.907969 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:05.907933    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"]
Sep 15 21:16:05.911335 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:05.911315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"
Sep 15 21:16:05.913412 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:05.913391    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 21:16:05.913796 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:05.913777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-m94pp\""
Sep 15 21:16:05.913796 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:05.913784    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:16:05.920650 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:05.920618    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"]
Sep 15 21:16:05.989175 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:05.989145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4w4\" (UniqueName: \"kubernetes.io/projected/d87bda14-7549-4811-9a53-e67940182bc2-kube-api-access-js4w4\") pod \"openshift-lws-operator-fd8ccff4c-kzszk\" (UID: \"d87bda14-7549-4811-9a53-e67940182bc2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"
Sep 15 21:16:05.989294 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:05.989188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87bda14-7549-4811-9a53-e67940182bc2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzszk\" (UID: \"d87bda14-7549-4811-9a53-e67940182bc2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"
Sep 15 21:16:06.089923 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:06.089882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4w4\" (UniqueName: \"kubernetes.io/projected/d87bda14-7549-4811-9a53-e67940182bc2-kube-api-access-js4w4\") pod \"openshift-lws-operator-fd8ccff4c-kzszk\" (UID: \"d87bda14-7549-4811-9a53-e67940182bc2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"
Sep 15 21:16:06.090032 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:06.089937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87bda14-7549-4811-9a53-e67940182bc2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzszk\" (UID: \"d87bda14-7549-4811-9a53-e67940182bc2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"
Sep 15 21:16:06.090264 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:06.090240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87bda14-7549-4811-9a53-e67940182bc2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzszk\" (UID: \"d87bda14-7549-4811-9a53-e67940182bc2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"
Sep 15 21:16:06.100949 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:06.100926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4w4\" (UniqueName: \"kubernetes.io/projected/d87bda14-7549-4811-9a53-e67940182bc2-kube-api-access-js4w4\") pod \"openshift-lws-operator-fd8ccff4c-kzszk\" (UID: \"d87bda14-7549-4811-9a53-e67940182bc2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"
Sep 15 21:16:06.221409 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:06.221386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"
Sep 15 21:16:06.490147 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:06.490069    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk"]
Sep 15 21:16:06.492987 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:16:06.492959    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd87bda14_7549_4811_9a53_e67940182bc2.slice/crio-17450714f542fc63687e91a7207842ecc0893b311dcc96ba81e20ee9df7fe322 WatchSource:0}: Error finding container 17450714f542fc63687e91a7207842ecc0893b311dcc96ba81e20ee9df7fe322: Status 404 returned error can't find the container with id 17450714f542fc63687e91a7207842ecc0893b311dcc96ba81e20ee9df7fe322
Sep 15 21:16:06.664520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:06.664490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk" event={"ID":"d87bda14-7549-4811-9a53-e67940182bc2","Type":"ContainerStarted","Data":"17450714f542fc63687e91a7207842ecc0893b311dcc96ba81e20ee9df7fe322"}
Sep 15 21:16:09.679841 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:09.679801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk" event={"ID":"d87bda14-7549-4811-9a53-e67940182bc2","Type":"ContainerStarted","Data":"a65bd9bed1eb27d5a4cd7929469a6614452de0d1d9e0ad07b90eb5a036743db9"}
Sep 15 21:16:09.703074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:09.703026    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzszk" podStartSLOduration=2.378532887 podStartE2EDuration="4.703010595s" totalImagesPullingTime="2.324477708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:09.701724788 +0000 UTC m=+488.301636896" watchObservedRunningTime="2026-09-15 21:16:09.703010595 +0000 UTC m=+488.302922703"
Sep 15 21:16:25.274231 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.274190    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"]
Sep 15 21:16:25.277924 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.277902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.281624 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.281598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 21:16:25.281789 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.281625    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 21:16:25.281931 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.281914    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 21:16:25.281978 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.281918    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 21:16:25.281978 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.281939    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d2xlj\""
Sep 15 21:16:25.312808 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.312785    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"]
Sep 15 21:16:25.444336 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.444310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-manifests\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.444510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.444357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-tmp\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.444510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.444390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.444510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.444470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zh5\" (UniqueName: \"kubernetes.io/projected/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-kube-api-access-74zh5\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.444510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.444492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.545867 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.545786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-tmp\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.545867 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.545824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.546076 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.545883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zh5\" (UniqueName: \"kubernetes.io/projected/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-kube-api-access-74zh5\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.546076 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.545902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.546076 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.545929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-manifests\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.546367 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.546333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-manifests\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.548333 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.548306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-tmp\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.548478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.548433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.548522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.548501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.556833 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.556809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zh5\" (UniqueName: \"kubernetes.io/projected/1588d5a6-75ff-4a81-adf0-3d94cebe87d6-kube-api-access-74zh5\") pod \"opendatahub-operator-controller-manager-546ffd48db-59vbp\" (UID: \"1588d5a6-75ff-4a81-adf0-3d94cebe87d6\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.587680 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.587654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:25.881677 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:25.881578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"]
Sep 15 21:16:25.884707 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:16:25.884675    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1588d5a6_75ff_4a81_adf0_3d94cebe87d6.slice/crio-ad4a9d64f7d6c690e74ca37e11f7b6f3865fd13c2bb49c7cc271fd881cc7a064 WatchSource:0}: Error finding container ad4a9d64f7d6c690e74ca37e11f7b6f3865fd13c2bb49c7cc271fd881cc7a064: Status 404 returned error can't find the container with id ad4a9d64f7d6c690e74ca37e11f7b6f3865fd13c2bb49c7cc271fd881cc7a064
Sep 15 21:16:26.750512 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:26.750432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp" event={"ID":"1588d5a6-75ff-4a81-adf0-3d94cebe87d6","Type":"ContainerStarted","Data":"ad4a9d64f7d6c690e74ca37e11f7b6f3865fd13c2bb49c7cc271fd881cc7a064"}
Sep 15 21:16:28.762688 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:28.762654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp" event={"ID":"1588d5a6-75ff-4a81-adf0-3d94cebe87d6","Type":"ContainerStarted","Data":"ea99e97041c4504c7a064ebac09d729eba67979489073c0462fcce364093d360"}
Sep 15 21:16:30.773069 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:30.773032    2563 generic.go:356] "Generic (PLEG): container finished" podID="1588d5a6-75ff-4a81-adf0-3d94cebe87d6" containerID="ea99e97041c4504c7a064ebac09d729eba67979489073c0462fcce364093d360" exitCode=0
Sep 15 21:16:30.773489 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:30.773085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp" event={"ID":"1588d5a6-75ff-4a81-adf0-3d94cebe87d6","Type":"ContainerDied","Data":"ea99e97041c4504c7a064ebac09d729eba67979489073c0462fcce364093d360"}
Sep 15 21:16:31.779859 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:31.779822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp" event={"ID":"1588d5a6-75ff-4a81-adf0-3d94cebe87d6","Type":"ContainerStarted","Data":"97678332c7eb927861ae061f312fac5e34d5ec877afcdb2eb7d40c4f1dc7c6c8"}
Sep 15 21:16:31.780401 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:31.780051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:31.803397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:31.803324    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp" podStartSLOduration=2.555907681 podStartE2EDuration="6.803311374s" totalImagesPullingTime="3.161294508s" totalInitContainerRuntime="1.086109185s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:31.802328509 +0000 UTC m=+510.402240619" watchObservedRunningTime="2026-09-15 21:16:31.803311374 +0000 UTC m=+510.403223484"
Sep 15 21:16:42.787095 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:42.787064    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-59vbp"
Sep 15 21:16:45.222115 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.222082    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"]
Sep 15 21:16:45.239165 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.239141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.258338 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.258312    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 21:16:45.258646 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.258629    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 21:16:45.258954 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.258936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 21:16:45.259047 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.258974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 21:16:45.259047 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.259002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zx8xv\""
Sep 15 21:16:45.284673 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.284644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"]
Sep 15 21:16:45.284915 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.284897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m88r\" (UniqueName: \"kubernetes.io/projected/1175e460-a0fb-448a-b69c-0566d4f36878-kube-api-access-7m88r\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.284967 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.284935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1175e460-a0fb-448a-b69c-0566d4f36878-tls-certs\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.285008 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.284970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1175e460-a0fb-448a-b69c-0566d4f36878-tmp\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.385426 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.385389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m88r\" (UniqueName: \"kubernetes.io/projected/1175e460-a0fb-448a-b69c-0566d4f36878-kube-api-access-7m88r\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.385616 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.385437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1175e460-a0fb-448a-b69c-0566d4f36878-tls-certs\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.385616 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.385506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1175e460-a0fb-448a-b69c-0566d4f36878-tmp\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.388018 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.387992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1175e460-a0fb-448a-b69c-0566d4f36878-tmp\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.388267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.388249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1175e460-a0fb-448a-b69c-0566d4f36878-tls-certs\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.415629 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.415602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m88r\" (UniqueName: \"kubernetes.io/projected/1175e460-a0fb-448a-b69c-0566d4f36878-kube-api-access-7m88r\") pod \"kube-auth-proxy-695d9b6867-77rmc\" (UID: \"1175e460-a0fb-448a-b69c-0566d4f36878\") " pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.548700 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.548608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"
Sep 15 21:16:45.843598 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:45.843572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-695d9b6867-77rmc"]
Sep 15 21:16:45.846117 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:16:45.846087    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1175e460_a0fb_448a_b69c_0566d4f36878.slice/crio-321dd5e236015f927df28775cb87d60497b4ba5c78cc626fa33a4c871ce4cce8 WatchSource:0}: Error finding container 321dd5e236015f927df28775cb87d60497b4ba5c78cc626fa33a4c871ce4cce8: Status 404 returned error can't find the container with id 321dd5e236015f927df28775cb87d60497b4ba5c78cc626fa33a4c871ce4cce8
Sep 15 21:16:46.848994 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:46.848904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc" event={"ID":"1175e460-a0fb-448a-b69c-0566d4f36878","Type":"ContainerStarted","Data":"321dd5e236015f927df28775cb87d60497b4ba5c78cc626fa33a4c871ce4cce8"}
Sep 15 21:16:47.750288 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.750257    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"]
Sep 15 21:16:47.753797 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.753774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.758005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.757881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:16:47.758005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.757896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 21:16:47.758005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.757942    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 21:16:47.758346 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.758304    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 21:16:47.758346 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.758316    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-p2jqf\""
Sep 15 21:16:47.784291 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.784263    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"]
Sep 15 21:16:47.809127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.809093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.809127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.809132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.809365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.809157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd39faf-600a-49a2-a590-94b8722b0f14-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.809365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.809207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdgz2\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-kube-api-access-sdgz2\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.809365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.809236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.809365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.809301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.809365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.809342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.910661 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.910615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.911127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.910670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.911127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.910703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd39faf-600a-49a2-a590-94b8722b0f14-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.911127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.910822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdgz2\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-kube-api-access-sdgz2\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.911127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.910868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.911127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.910929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.911127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.911061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.911539 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.911356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.918473 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.915602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.918473 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.916384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.924007 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.923887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdgz2\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-kube-api-access-sdgz2\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.924007 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.923887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd39faf-600a-49a2-a590-94b8722b0f14-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.924827 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.924783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:47.926251 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:47.926209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6mkln\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:48.066589 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:48.066508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:48.860347 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:48.860314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc" event={"ID":"1175e460-a0fb-448a-b69c-0566d4f36878","Type":"ContainerStarted","Data":"edce1fcb06ef8f030f6ade9c0175d5db1192eb3e94e3d780f8b24828bd5db59a"}
Sep 15 21:16:48.879106 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:48.879061    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-695d9b6867-77rmc" podStartSLOduration=0.98404602 podStartE2EDuration="3.879047847s" totalImagesPullingTime="2.895001827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:48.877998675 +0000 UTC m=+527.477910784" watchObservedRunningTime="2026-09-15 21:16:48.879047847 +0000 UTC m=+527.478959956"
Sep 15 21:16:48.959419 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:48.959294    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"]
Sep 15 21:16:48.962037 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:16:48.962007    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd39faf_600a_49a2_a590_94b8722b0f14.slice/crio-1e7e33d5a743739433dae6b682ecc5bd2dad59bf2ac283e0f5820edbe100a09e WatchSource:0}: Error finding container 1e7e33d5a743739433dae6b682ecc5bd2dad59bf2ac283e0f5820edbe100a09e: Status 404 returned error can't find the container with id 1e7e33d5a743739433dae6b682ecc5bd2dad59bf2ac283e0f5820edbe100a09e
Sep 15 21:16:49.868080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:49.868036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln" event={"ID":"7fd39faf-600a-49a2-a590-94b8722b0f14","Type":"ContainerStarted","Data":"1e7e33d5a743739433dae6b682ecc5bd2dad59bf2ac283e0f5820edbe100a09e"}
Sep 15 21:16:51.236993 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:51.236954    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:16:51.237288 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:51.237021    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:16:51.882287 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:51.882253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln" event={"ID":"7fd39faf-600a-49a2-a590-94b8722b0f14","Type":"ContainerStarted","Data":"f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768"}
Sep 15 21:16:51.882493 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:51.882384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:16:51.905199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:51.905146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln" podStartSLOduration=2.632551935 podStartE2EDuration="4.905088478s" totalImagesPullingTime="2.272536543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:51.903074115 +0000 UTC m=+530.502986225" watchObservedRunningTime="2026-09-15 21:16:51.905088478 +0000 UTC m=+530.505000588"
Sep 15 21:16:52.888151 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:16:52.888077    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:17:02.867253 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.867218    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"]
Sep 15 21:17:02.873670 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.873651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:02.889533 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.889509    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"]
Sep 15 21:17:02.942213 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.942183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:02.942381 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.942229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:02.942381 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.942246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:02.942381 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.942320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:02.942381 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.942367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:02.942546 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.942437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2s64\" (UniqueName: \"kubernetes.io/projected/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-kube-api-access-d2s64\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:02.942546 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:02.942494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.043787 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.043755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.043939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.043804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.043939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.043821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.043939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.043857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.043939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.043890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.043939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.043932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2s64\" (UniqueName: \"kubernetes.io/projected/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-kube-api-access-d2s64\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.044214 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.043969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.044629 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.044602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.046408 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.046383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.046654 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.046624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.046882 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.046863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.046957 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.046928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.062723 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.062698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.076695 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.076670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2s64\" (UniqueName: \"kubernetes.io/projected/f9d540ad-1119-4ee4-bdce-a236e94c6ffe-kube-api-access-d2s64\") pod \"istiod-openshift-gateway-5677ccb49b-8rwkn\" (UID: \"f9d540ad-1119-4ee4-bdce-a236e94c6ffe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.182876 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.182778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.463564 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.463536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"]
Sep 15 21:17:03.465848 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:17:03.465821    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d540ad_1119_4ee4_bdce_a236e94c6ffe.slice/crio-165916b91873363bcf44888aab0acdc6054b31ed53cc6848e4f5890b2959ca76 WatchSource:0}: Error finding container 165916b91873363bcf44888aab0acdc6054b31ed53cc6848e4f5890b2959ca76: Status 404 returned error can't find the container with id 165916b91873363bcf44888aab0acdc6054b31ed53cc6848e4f5890b2959ca76
Sep 15 21:17:03.468036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.468004    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:17:03.468102 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.468065    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:17:03.937071 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.936884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn" event={"ID":"f9d540ad-1119-4ee4-bdce-a236e94c6ffe","Type":"ContainerStarted","Data":"0ed579c3f2b928d2b72ea9f900c730cc18cf1a19e8504ef2ab54b9758974c3f8"}
Sep 15 21:17:03.937071 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.936942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn" event={"ID":"f9d540ad-1119-4ee4-bdce-a236e94c6ffe","Type":"ContainerStarted","Data":"165916b91873363bcf44888aab0acdc6054b31ed53cc6848e4f5890b2959ca76"}
Sep 15 21:17:03.937071 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.937005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:03.966852 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:03.966799    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn" podStartSLOduration=1.966781468 podStartE2EDuration="1.966781468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:03.965629615 +0000 UTC m=+542.565541727" watchObservedRunningTime="2026-09-15 21:17:03.966781468 +0000 UTC m=+542.566693578"
Sep 15 21:17:04.943210 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:04.943179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8rwkn"
Sep 15 21:17:05.066634 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.066605    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"]
Sep 15 21:17:05.066820 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.066785    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln" podUID="7fd39faf-600a-49a2-a590-94b8722b0f14" containerName="discovery" containerID="cri-o://f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768" gracePeriod=30
Sep 15 21:17:05.315641 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.315619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:17:05.368384 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.368353    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdgz2\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-kube-api-access-sdgz2\") pod \"7fd39faf-600a-49a2-a590-94b8722b0f14\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") "
Sep 15 21:17:05.368572 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.368398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-token\") pod \"7fd39faf-600a-49a2-a590-94b8722b0f14\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") "
Sep 15 21:17:05.368572 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.368423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd39faf-600a-49a2-a590-94b8722b0f14-local-certs\") pod \"7fd39faf-600a-49a2-a590-94b8722b0f14\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") "
Sep 15 21:17:05.368572 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.368536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-cacerts\") pod \"7fd39faf-600a-49a2-a590-94b8722b0f14\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") "
Sep 15 21:17:05.368741 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.368580    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-kubeconfig\") pod \"7fd39faf-600a-49a2-a590-94b8722b0f14\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") "
Sep 15 21:17:05.368741 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.368623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-ca-configmap\") pod \"7fd39faf-600a-49a2-a590-94b8722b0f14\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") "
Sep 15 21:17:05.368741 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.368650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-dns-cert\") pod \"7fd39faf-600a-49a2-a590-94b8722b0f14\" (UID: \"7fd39faf-600a-49a2-a590-94b8722b0f14\") "
Sep 15 21:17:05.370043 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.370009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7fd39faf-600a-49a2-a590-94b8722b0f14" (UID: "7fd39faf-600a-49a2-a590-94b8722b0f14"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:17:05.371570 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.371534    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-kube-api-access-sdgz2" (OuterVolumeSpecName: "kube-api-access-sdgz2") pod "7fd39faf-600a-49a2-a590-94b8722b0f14" (UID: "7fd39faf-600a-49a2-a590-94b8722b0f14"). InnerVolumeSpecName "kube-api-access-sdgz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:17:05.372021 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.371986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7fd39faf-600a-49a2-a590-94b8722b0f14" (UID: "7fd39faf-600a-49a2-a590-94b8722b0f14"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:17:05.372206 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.372175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7fd39faf-600a-49a2-a590-94b8722b0f14" (UID: "7fd39faf-600a-49a2-a590-94b8722b0f14"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:17:05.372206 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.372175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd39faf-600a-49a2-a590-94b8722b0f14-local-certs" (OuterVolumeSpecName: "local-certs") pod "7fd39faf-600a-49a2-a590-94b8722b0f14" (UID: "7fd39faf-600a-49a2-a590-94b8722b0f14"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:05.372300 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.372200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-cacerts" (OuterVolumeSpecName: "cacerts") pod "7fd39faf-600a-49a2-a590-94b8722b0f14" (UID: "7fd39faf-600a-49a2-a590-94b8722b0f14"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:17:05.373296 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.373277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fd39faf-600a-49a2-a590-94b8722b0f14" (UID: "7fd39faf-600a-49a2-a590-94b8722b0f14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:17:05.469689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.469643    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-cacerts\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:17:05.469689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.469683    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-kubeconfig\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:17:05.469689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.469696    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-ca-configmap\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:17:05.469689 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.469705    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-csr-dns-cert\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:17:05.469958 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.469714    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdgz2\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-kube-api-access-sdgz2\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:17:05.469958 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.469722    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd39faf-600a-49a2-a590-94b8722b0f14-istio-token\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:17:05.469958 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.469729    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd39faf-600a-49a2-a590-94b8722b0f14-local-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:17:05.947361 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.947332    2563 generic.go:356] "Generic (PLEG): container finished" podID="7fd39faf-600a-49a2-a590-94b8722b0f14" containerID="f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768" exitCode=0
Sep 15 21:17:05.947767 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.947382    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"
Sep 15 21:17:05.947767 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.947421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln" event={"ID":"7fd39faf-600a-49a2-a590-94b8722b0f14","Type":"ContainerDied","Data":"f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768"}
Sep 15 21:17:05.947767 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.947488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln" event={"ID":"7fd39faf-600a-49a2-a590-94b8722b0f14","Type":"ContainerDied","Data":"1e7e33d5a743739433dae6b682ecc5bd2dad59bf2ac283e0f5820edbe100a09e"}
Sep 15 21:17:05.947767 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.947509    2563 scope.go:117] "RemoveContainer" containerID="f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768"
Sep 15 21:17:05.956399 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.956384    2563 scope.go:117] "RemoveContainer" containerID="f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768"
Sep 15 21:17:05.956662 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:17:05.956638    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768\": container with ID starting with f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768 not found: ID does not exist" containerID="f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768"
Sep 15 21:17:05.956744 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.956671    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768"} err="failed to get container status \"f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768\": rpc error: code = NotFound desc = could not find container \"f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768\": container with ID starting with f1f5e8b92cafcd9ea513014fc4c14587eabbb10a5371ba489a39f2c7ca037768 not found: ID does not exist"
Sep 15 21:17:05.981087 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.981065    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"]
Sep 15 21:17:05.989501 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:05.989463    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6mkln"]
Sep 15 21:17:07.900256 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:17:07.900222    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd39faf-600a-49a2-a590-94b8722b0f14" path="/var/lib/kubelet/pods/7fd39faf-600a-49a2-a590-94b8722b0f14/volumes"
Sep 15 21:18:15.081884 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.081854    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"]
Sep 15 21:18:15.084230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.082205    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd39faf-600a-49a2-a590-94b8722b0f14" containerName="discovery"
Sep 15 21:18:15.084230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.082224    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd39faf-600a-49a2-a590-94b8722b0f14" containerName="discovery"
Sep 15 21:18:15.084230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.082303    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd39faf-600a-49a2-a590-94b8722b0f14" containerName="discovery"
Sep 15 21:18:15.085130 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.085116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:15.087593 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.087570    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qrkx8\""
Sep 15 21:18:15.088070 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.088057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:18:15.088125 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.088097    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:18:15.107419 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.107395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"]
Sep 15 21:18:15.260218 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.260182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d552f4-8e47-4fb1-8acb-97e469728c34-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" (UID: \"82d552f4-8e47-4fb1-8acb-97e469728c34\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:15.260393 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.260261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4kn\" (UniqueName: \"kubernetes.io/projected/82d552f4-8e47-4fb1-8acb-97e469728c34-kube-api-access-8l4kn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" (UID: \"82d552f4-8e47-4fb1-8acb-97e469728c34\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:15.361390 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.361288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4kn\" (UniqueName: \"kubernetes.io/projected/82d552f4-8e47-4fb1-8acb-97e469728c34-kube-api-access-8l4kn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" (UID: \"82d552f4-8e47-4fb1-8acb-97e469728c34\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:15.361390 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.361349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d552f4-8e47-4fb1-8acb-97e469728c34-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" (UID: \"82d552f4-8e47-4fb1-8acb-97e469728c34\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:15.361774 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.361754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d552f4-8e47-4fb1-8acb-97e469728c34-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" (UID: \"82d552f4-8e47-4fb1-8acb-97e469728c34\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:15.385343 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.385314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4kn\" (UniqueName: \"kubernetes.io/projected/82d552f4-8e47-4fb1-8acb-97e469728c34-kube-api-access-8l4kn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" (UID: \"82d552f4-8e47-4fb1-8acb-97e469728c34\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:15.395159 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.395135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:15.683998 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.683779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"]
Sep 15 21:18:15.687077 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:18:15.687044    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d552f4_8e47_4fb1_8acb_97e469728c34.slice/crio-952b1b8fb469b187d13af1c5b84f297acc5ea45c09081915b97c4fe177837689 WatchSource:0}: Error finding container 952b1b8fb469b187d13af1c5b84f297acc5ea45c09081915b97c4fe177837689: Status 404 returned error can't find the container with id 952b1b8fb469b187d13af1c5b84f297acc5ea45c09081915b97c4fe177837689
Sep 15 21:18:15.689748 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:15.689733    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:18:16.285091 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:16.285057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4" event={"ID":"82d552f4-8e47-4fb1-8acb-97e469728c34","Type":"ContainerStarted","Data":"952b1b8fb469b187d13af1c5b84f297acc5ea45c09081915b97c4fe177837689"}
Sep 15 21:18:19.801236 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:19.801200    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cbcv7"]
Sep 15 21:18:19.805972 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:19.805951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7"
Sep 15 21:18:19.808028 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:19.808000    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ftbbd\""
Sep 15 21:18:19.824109 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:19.824077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cbcv7"]
Sep 15 21:18:20.005164 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:20.005130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzkw\" (UniqueName: \"kubernetes.io/projected/933f7889-b755-4c04-912b-faf8ea7c52be-kube-api-access-jdzkw\") pod \"authorino-operator-76b478bd78-cbcv7\" (UID: \"933f7889-b755-4c04-912b-faf8ea7c52be\") " pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7"
Sep 15 21:18:20.106248 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:20.106153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzkw\" (UniqueName: \"kubernetes.io/projected/933f7889-b755-4c04-912b-faf8ea7c52be-kube-api-access-jdzkw\") pod \"authorino-operator-76b478bd78-cbcv7\" (UID: \"933f7889-b755-4c04-912b-faf8ea7c52be\") " pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7"
Sep 15 21:18:20.113901 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:20.113864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzkw\" (UniqueName: \"kubernetes.io/projected/933f7889-b755-4c04-912b-faf8ea7c52be-kube-api-access-jdzkw\") pod \"authorino-operator-76b478bd78-cbcv7\" (UID: \"933f7889-b755-4c04-912b-faf8ea7c52be\") " pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7"
Sep 15 21:18:20.117617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:20.117596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7"
Sep 15 21:18:20.310716 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:20.310675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4" event={"ID":"82d552f4-8e47-4fb1-8acb-97e469728c34","Type":"ContainerStarted","Data":"c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865"}
Sep 15 21:18:20.310883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:20.310802    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:20.337498 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:20.337431    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4" podStartSLOduration=1.123451694 podStartE2EDuration="5.337412254s" totalImagesPullingTime="4.21396056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:20.335911203 +0000 UTC m=+618.935823314" watchObservedRunningTime="2026-09-15 21:18:20.337412254 +0000 UTC m=+618.937324378"
Sep 15 21:18:20.392891 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:20.392810    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cbcv7"]
Sep 15 21:18:20.396540 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:18:20.396507    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933f7889_b755_4c04_912b_faf8ea7c52be.slice/crio-58348076e2ab4e5560a395a1b519860b564aca3d38bb2232663b0ed1bd471dd7 WatchSource:0}: Error finding container 58348076e2ab4e5560a395a1b519860b564aca3d38bb2232663b0ed1bd471dd7: Status 404 returned error can't find the container with id 58348076e2ab4e5560a395a1b519860b564aca3d38bb2232663b0ed1bd471dd7
Sep 15 21:18:21.317781 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:21.317735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7" event={"ID":"933f7889-b755-4c04-912b-faf8ea7c52be","Type":"ContainerStarted","Data":"58348076e2ab4e5560a395a1b519860b564aca3d38bb2232663b0ed1bd471dd7"}
Sep 15 21:18:22.324746 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:22.324713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7" event={"ID":"933f7889-b755-4c04-912b-faf8ea7c52be","Type":"ContainerStarted","Data":"b77e4fa76a5c4e5fad905dd8118209fb4e4eb8aa70a07abcdcab07e59e2b6ebb"}
Sep 15 21:18:22.325126 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:22.324787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7"
Sep 15 21:18:22.358518 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:22.358465    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7" podStartSLOduration=1.855383153 podStartE2EDuration="3.358425208s" totalImagesPullingTime="1.503042055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:22.357380818 +0000 UTC m=+620.957292928" watchObservedRunningTime="2026-09-15 21:18:22.358425208 +0000 UTC m=+620.958337319"
Sep 15 21:18:31.320565 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:31.320528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:33.331694 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.331661    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-cbcv7"
Sep 15 21:18:33.555524 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.555487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"]
Sep 15 21:18:33.555724 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.555678    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4" podUID="82d552f4-8e47-4fb1-8acb-97e469728c34" containerName="manager" containerID="cri-o://c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865" gracePeriod=2
Sep 15 21:18:33.575374 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.575345    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"]
Sep 15 21:18:33.579145 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:18:33.579114    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"82d552f4-8e47-4fb1-8acb-97e469728c34\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\": pods \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:33.589727 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.589656    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"]
Sep 15 21:18:33.590238 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.590222    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d552f4-8e47-4fb1-8acb-97e469728c34" containerName="manager"
Sep 15 21:18:33.590297 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.590243    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d552f4-8e47-4fb1-8acb-97e469728c34" containerName="manager"
Sep 15 21:18:33.590375 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.590363    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d552f4-8e47-4fb1-8acb-97e469728c34" containerName="manager"
Sep 15 21:18:33.593814 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.593791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:33.595698 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:18:33.595668    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" is forbidden: User \"system:node:ip-10-0-140-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-113.ec2.internal' and this object" podUID="82d552f4-8e47-4fb1-8acb-97e469728c34" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:33.612028 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.611997    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"]
Sep 15 21:18:33.618428 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.618396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8qp\" (UniqueName: \"kubernetes.io/projected/f2144e5c-1cb6-4bd0-b454-236c6a433511-kube-api-access-4w8qp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-65mc9\" (UID: \"f2144e5c-1cb6-4bd0-b454-236c6a433511\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:33.618615 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.618468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2144e5c-1cb6-4bd0-b454-236c6a433511-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-65mc9\" (UID: \"f2144e5c-1cb6-4bd0-b454-236c6a433511\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:33.623944 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.623913    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"]
Sep 15 21:18:33.627416 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.627392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"
Sep 15 21:18:33.630338 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.630318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jngc9\""
Sep 15 21:18:33.642007 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.641964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"]
Sep 15 21:18:33.645064 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:18:33.645031    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" is forbidden: User \"system:node:ip-10-0-140-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-113.ec2.internal' and this object" podUID="82d552f4-8e47-4fb1-8acb-97e469728c34" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:33.719757 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.719715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2144e5c-1cb6-4bd0-b454-236c6a433511-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-65mc9\" (UID: \"f2144e5c-1cb6-4bd0-b454-236c6a433511\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:33.719944 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.719774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qls\" (UniqueName: \"kubernetes.io/projected/37953e7e-906e-45b2-9087-2d143b8f7487-kube-api-access-q8qls\") pod \"limitador-operator-controller-manager-74495c69f8-tncjr\" (UID: \"37953e7e-906e-45b2-9087-2d143b8f7487\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"
Sep 15 21:18:33.719944 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.719926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8qp\" (UniqueName: \"kubernetes.io/projected/f2144e5c-1cb6-4bd0-b454-236c6a433511-kube-api-access-4w8qp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-65mc9\" (UID: \"f2144e5c-1cb6-4bd0-b454-236c6a433511\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:33.720168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.720148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2144e5c-1cb6-4bd0-b454-236c6a433511-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-65mc9\" (UID: \"f2144e5c-1cb6-4bd0-b454-236c6a433511\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:33.729119 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.729088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8qp\" (UniqueName: \"kubernetes.io/projected/f2144e5c-1cb6-4bd0-b454-236c6a433511-kube-api-access-4w8qp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-65mc9\" (UID: \"f2144e5c-1cb6-4bd0-b454-236c6a433511\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:33.800386 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.800359    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:33.803263 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:18:33.803237    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f9qc4\" is forbidden: User \"system:node:ip-10-0-140-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-113.ec2.internal' and this object" podUID="82d552f4-8e47-4fb1-8acb-97e469728c34" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:33.820776 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.820740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qls\" (UniqueName: \"kubernetes.io/projected/37953e7e-906e-45b2-9087-2d143b8f7487-kube-api-access-q8qls\") pod \"limitador-operator-controller-manager-74495c69f8-tncjr\" (UID: \"37953e7e-906e-45b2-9087-2d143b8f7487\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"
Sep 15 21:18:33.839113 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.839078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qls\" (UniqueName: \"kubernetes.io/projected/37953e7e-906e-45b2-9087-2d143b8f7487-kube-api-access-q8qls\") pod \"limitador-operator-controller-manager-74495c69f8-tncjr\" (UID: \"37953e7e-906e-45b2-9087-2d143b8f7487\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"
Sep 15 21:18:33.922115 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.922023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d552f4-8e47-4fb1-8acb-97e469728c34-extensions-socket-volume\") pod \"82d552f4-8e47-4fb1-8acb-97e469728c34\" (UID: \"82d552f4-8e47-4fb1-8acb-97e469728c34\") "
Sep 15 21:18:33.922115 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.922091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l4kn\" (UniqueName: \"kubernetes.io/projected/82d552f4-8e47-4fb1-8acb-97e469728c34-kube-api-access-8l4kn\") pod \"82d552f4-8e47-4fb1-8acb-97e469728c34\" (UID: \"82d552f4-8e47-4fb1-8acb-97e469728c34\") "
Sep 15 21:18:33.922554 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.922531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d552f4-8e47-4fb1-8acb-97e469728c34-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "82d552f4-8e47-4fb1-8acb-97e469728c34" (UID: "82d552f4-8e47-4fb1-8acb-97e469728c34"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:33.924469 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.924429    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d552f4-8e47-4fb1-8acb-97e469728c34-kube-api-access-8l4kn" (OuterVolumeSpecName: "kube-api-access-8l4kn") pod "82d552f4-8e47-4fb1-8acb-97e469728c34" (UID: "82d552f4-8e47-4fb1-8acb-97e469728c34"). InnerVolumeSpecName "kube-api-access-8l4kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:33.934602 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.934575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:33.943331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:33.943294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"
Sep 15 21:18:34.023055 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.023022    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d552f4-8e47-4fb1-8acb-97e469728c34-extensions-socket-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:18:34.023055 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.023055    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4kn\" (UniqueName: \"kubernetes.io/projected/82d552f4-8e47-4fb1-8acb-97e469728c34-kube-api-access-8l4kn\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:18:34.241790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.241675    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"]
Sep 15 21:18:34.247820 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.247792    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"]
Sep 15 21:18:34.248285 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:18:34.248264    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2144e5c_1cb6_4bd0_b454_236c6a433511.slice/crio-a0079a1fe0d83f872f9c4b6dac7c11d1faea1d1fe7aae99a28c010a6afeac2e0 WatchSource:0}: Error finding container a0079a1fe0d83f872f9c4b6dac7c11d1faea1d1fe7aae99a28c010a6afeac2e0: Status 404 returned error can't find the container with id a0079a1fe0d83f872f9c4b6dac7c11d1faea1d1fe7aae99a28c010a6afeac2e0
Sep 15 21:18:34.248986 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:18:34.248965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37953e7e_906e_45b2_9087_2d143b8f7487.slice/crio-d8c232401d9937cae9208f9f5253f56e3e0cc416dc93a8ccf8ca02fa3d285936 WatchSource:0}: Error finding container d8c232401d9937cae9208f9f5253f56e3e0cc416dc93a8ccf8ca02fa3d285936: Status 404 returned error can't find the container with id d8c232401d9937cae9208f9f5253f56e3e0cc416dc93a8ccf8ca02fa3d285936
Sep 15 21:18:34.388487 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.388436    2563 generic.go:356] "Generic (PLEG): container finished" podID="82d552f4-8e47-4fb1-8acb-97e469728c34" containerID="c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865" exitCode=0
Sep 15 21:18:34.389020 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.388538    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f9qc4"
Sep 15 21:18:34.389020 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.388569    2563 scope.go:117] "RemoveContainer" containerID="c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865"
Sep 15 21:18:34.390622 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.390576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr" event={"ID":"37953e7e-906e-45b2-9087-2d143b8f7487","Type":"ContainerStarted","Data":"d8c232401d9937cae9208f9f5253f56e3e0cc416dc93a8ccf8ca02fa3d285936"}
Sep 15 21:18:34.393104 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.393046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9" event={"ID":"f2144e5c-1cb6-4bd0-b454-236c6a433511","Type":"ContainerStarted","Data":"1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30"}
Sep 15 21:18:34.393291 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.393122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9" event={"ID":"f2144e5c-1cb6-4bd0-b454-236c6a433511","Type":"ContainerStarted","Data":"a0079a1fe0d83f872f9c4b6dac7c11d1faea1d1fe7aae99a28c010a6afeac2e0"}
Sep 15 21:18:34.393291 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.393179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:34.400320 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.400290    2563 scope.go:117] "RemoveContainer" containerID="c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865"
Sep 15 21:18:34.400804 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:18:34.400774    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865\": container with ID starting with c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865 not found: ID does not exist" containerID="c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865"
Sep 15 21:18:34.400895 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.400822    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865"} err="failed to get container status \"c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865\": rpc error: code = NotFound desc = could not find container \"c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865\": container with ID starting with c4f5e17ed51ce955759dd87e74e2d05e9c081ce33a9514b9276b6cea57af3865 not found: ID does not exist"
Sep 15 21:18:34.417558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.417498    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9" podStartSLOduration=1.417478076 podStartE2EDuration="1.417478076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:34.41516507 +0000 UTC m=+633.015077179" watchObservedRunningTime="2026-09-15 21:18:34.417478076 +0000 UTC m=+633.017390177"
Sep 15 21:18:34.703775 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.703739    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"]
Sep 15 21:18:34.708520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.708496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:34.725761 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.725732    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"]
Sep 15 21:18:34.830518 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.830413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf6c48f7-b36c-4226-9105-848931709f56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pbhx6\" (UID: \"cf6c48f7-b36c-4226-9105-848931709f56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:34.830685 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.830544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zps\" (UniqueName: \"kubernetes.io/projected/cf6c48f7-b36c-4226-9105-848931709f56-kube-api-access-28zps\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pbhx6\" (UID: \"cf6c48f7-b36c-4226-9105-848931709f56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:34.931384 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.931335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf6c48f7-b36c-4226-9105-848931709f56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pbhx6\" (UID: \"cf6c48f7-b36c-4226-9105-848931709f56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:34.931582 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.931426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zps\" (UniqueName: \"kubernetes.io/projected/cf6c48f7-b36c-4226-9105-848931709f56-kube-api-access-28zps\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pbhx6\" (UID: \"cf6c48f7-b36c-4226-9105-848931709f56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:34.931834 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.931808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf6c48f7-b36c-4226-9105-848931709f56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pbhx6\" (UID: \"cf6c48f7-b36c-4226-9105-848931709f56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:34.942522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:34.942478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zps\" (UniqueName: \"kubernetes.io/projected/cf6c48f7-b36c-4226-9105-848931709f56-kube-api-access-28zps\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pbhx6\" (UID: \"cf6c48f7-b36c-4226-9105-848931709f56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:35.019003 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:35.018966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:35.310912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:35.310878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"]
Sep 15 21:18:35.519049 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:18:35.519010    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6c48f7_b36c_4226_9105_848931709f56.slice/crio-b8c77ca5e59d9985ec640efae4334e314ae70e7e117f8a3155ce68993dde34de WatchSource:0}: Error finding container b8c77ca5e59d9985ec640efae4334e314ae70e7e117f8a3155ce68993dde34de: Status 404 returned error can't find the container with id b8c77ca5e59d9985ec640efae4334e314ae70e7e117f8a3155ce68993dde34de
Sep 15 21:18:35.903537 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:35.903434    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d552f4-8e47-4fb1-8acb-97e469728c34" path="/var/lib/kubelet/pods/82d552f4-8e47-4fb1-8acb-97e469728c34/volumes"
Sep 15 21:18:36.408086 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:36.408050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr" event={"ID":"37953e7e-906e-45b2-9087-2d143b8f7487","Type":"ContainerStarted","Data":"526cac34f64274bd33dd9f86148bb400758cf26841e33ab378ebd390b61753e2"}
Sep 15 21:18:36.408280 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:36.408255    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"
Sep 15 21:18:36.409904 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:36.409873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6" event={"ID":"cf6c48f7-b36c-4226-9105-848931709f56","Type":"ContainerStarted","Data":"d6c66a1333c0ad7dda85513c9c4bd31d581401ec5bca87257d15e062f8f8eeec"}
Sep 15 21:18:36.409904 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:36.409909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6" event={"ID":"cf6c48f7-b36c-4226-9105-848931709f56","Type":"ContainerStarted","Data":"b8c77ca5e59d9985ec640efae4334e314ae70e7e117f8a3155ce68993dde34de"}
Sep 15 21:18:36.410104 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:36.409936    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:36.427487 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:36.427414    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr" podStartSLOduration=2.096845265 podStartE2EDuration="3.427398427s" totalImagesPullingTime="1.330553162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:36.42608059 +0000 UTC m=+635.025992699" watchObservedRunningTime="2026-09-15 21:18:36.427398427 +0000 UTC m=+635.027310538"
Sep 15 21:18:36.445228 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:36.445172    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6" podStartSLOduration=2.445155082 podStartE2EDuration="2.445155082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:36.44417075 +0000 UTC m=+635.044082863" watchObservedRunningTime="2026-09-15 21:18:36.445155082 +0000 UTC m=+635.045067192"
Sep 15 21:18:45.403643 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:45.403604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:47.416745 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.416712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:18:47.417209 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.416947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tncjr"
Sep 15 21:18:47.481895 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.481864    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"]
Sep 15 21:18:47.482116 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.482081    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9" podUID="f2144e5c-1cb6-4bd0-b454-236c6a433511" containerName="manager" containerID="cri-o://1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30" gracePeriod=10
Sep 15 21:18:47.747069 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.747041    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:47.847501 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.847419    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2144e5c-1cb6-4bd0-b454-236c6a433511-extensions-socket-volume\") pod \"f2144e5c-1cb6-4bd0-b454-236c6a433511\" (UID: \"f2144e5c-1cb6-4bd0-b454-236c6a433511\") "
Sep 15 21:18:47.847771 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.847572    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w8qp\" (UniqueName: \"kubernetes.io/projected/f2144e5c-1cb6-4bd0-b454-236c6a433511-kube-api-access-4w8qp\") pod \"f2144e5c-1cb6-4bd0-b454-236c6a433511\" (UID: \"f2144e5c-1cb6-4bd0-b454-236c6a433511\") "
Sep 15 21:18:47.847906 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.847880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2144e5c-1cb6-4bd0-b454-236c6a433511-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f2144e5c-1cb6-4bd0-b454-236c6a433511" (UID: "f2144e5c-1cb6-4bd0-b454-236c6a433511"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:47.850262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.850223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2144e5c-1cb6-4bd0-b454-236c6a433511-kube-api-access-4w8qp" (OuterVolumeSpecName: "kube-api-access-4w8qp") pod "f2144e5c-1cb6-4bd0-b454-236c6a433511" (UID: "f2144e5c-1cb6-4bd0-b454-236c6a433511"). InnerVolumeSpecName "kube-api-access-4w8qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:47.948866 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.948831    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8qp\" (UniqueName: \"kubernetes.io/projected/f2144e5c-1cb6-4bd0-b454-236c6a433511-kube-api-access-4w8qp\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:18:47.948866 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:47.948861    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2144e5c-1cb6-4bd0-b454-236c6a433511-extensions-socket-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:18:48.475278 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.475244    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2144e5c-1cb6-4bd0-b454-236c6a433511" containerID="1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30" exitCode=0
Sep 15 21:18:48.475764 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.475291    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"
Sep 15 21:18:48.475764 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.475311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9" event={"ID":"f2144e5c-1cb6-4bd0-b454-236c6a433511","Type":"ContainerDied","Data":"1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30"}
Sep 15 21:18:48.475764 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.475367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9" event={"ID":"f2144e5c-1cb6-4bd0-b454-236c6a433511","Type":"ContainerDied","Data":"a0079a1fe0d83f872f9c4b6dac7c11d1faea1d1fe7aae99a28c010a6afeac2e0"}
Sep 15 21:18:48.475764 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.475401    2563 scope.go:117] "RemoveContainer" containerID="1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30"
Sep 15 21:18:48.485220 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.484946    2563 scope.go:117] "RemoveContainer" containerID="1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30"
Sep 15 21:18:48.485320 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:18:48.485275    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30\": container with ID starting with 1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30 not found: ID does not exist" containerID="1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30"
Sep 15 21:18:48.485320 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.485302    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30"} err="failed to get container status \"1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30\": rpc error: code = NotFound desc = could not find container \"1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30\": container with ID starting with 1ec10f9714e2608982eb290d88d6a9fc45feb7093ae248cf6bcb32fdae578e30 not found: ID does not exist"
Sep 15 21:18:48.524440 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.524392    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"]
Sep 15 21:18:48.534038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:48.533999    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-65mc9"]
Sep 15 21:18:49.900941 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:18:49.900909    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2144e5c-1cb6-4bd0-b454-236c6a433511" path="/var/lib/kubelet/pods/f2144e5c-1cb6-4bd0-b454-236c6a433511/volumes"
Sep 15 21:19:46.858286 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.858250    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65b5854d44-5ll6l"]
Sep 15 21:19:46.858833 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.858702    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2144e5c-1cb6-4bd0-b454-236c6a433511" containerName="manager"
Sep 15 21:19:46.858833 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.858716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2144e5c-1cb6-4bd0-b454-236c6a433511" containerName="manager"
Sep 15 21:19:46.858833 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.858779    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2144e5c-1cb6-4bd0-b454-236c6a433511" containerName="manager"
Sep 15 21:19:46.862166 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.862144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:46.864384 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.864347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:19:46.864384 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.864383    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:19:46.865007 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.864992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wvswp\""
Sep 15 21:19:46.865087 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.865026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:19:46.872213 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.872182    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b5854d44-5ll6l"]
Sep 15 21:19:46.963880 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.963842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l947b\" (UniqueName: \"kubernetes.io/projected/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-kube-api-access-l947b\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:46.964079 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.963908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-webhook-cert\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:46.964079 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:46.963995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-metrics-certs\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:47.064933 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.064896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-metrics-certs\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:47.064933 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.064947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l947b\" (UniqueName: \"kubernetes.io/projected/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-kube-api-access-l947b\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:47.065414 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.064999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-webhook-cert\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:47.067728 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.067696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-webhook-cert\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:47.067847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.067730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-metrics-certs\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:47.074834 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.074800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l947b\" (UniqueName: \"kubernetes.io/projected/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-kube-api-access-l947b\") pod \"maas-controller-65b5854d44-5ll6l\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") " pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:47.174693 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.174594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:47.451815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.451792    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b5854d44-5ll6l"]
Sep 15 21:19:47.454576 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:19:47.454538    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa60b90f_dc4c_4bfb_9f7f_eed152473ba2.slice/crio-bc6cf421ae44f5513212f0c6ef738b7adc2731eace85a945a3f1a8894b0694dc WatchSource:0}: Error finding container bc6cf421ae44f5513212f0c6ef738b7adc2731eace85a945a3f1a8894b0694dc: Status 404 returned error can't find the container with id bc6cf421ae44f5513212f0c6ef738b7adc2731eace85a945a3f1a8894b0694dc
Sep 15 21:19:47.770112 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:47.770077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-5ll6l" event={"ID":"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2","Type":"ContainerStarted","Data":"bc6cf421ae44f5513212f0c6ef738b7adc2731eace85a945a3f1a8894b0694dc"}
Sep 15 21:19:49.792387 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:49.792341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-5ll6l" event={"ID":"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2","Type":"ContainerStarted","Data":"963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa"}
Sep 15 21:19:49.792819 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:49.792488    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:19:49.812440 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:19:49.812382    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65b5854d44-5ll6l" podStartSLOduration=1.564755589 podStartE2EDuration="3.812364231s" totalImagesPullingTime="2.247608642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:49.811631389 +0000 UTC m=+708.411543498" watchObservedRunningTime="2026-09-15 21:19:49.812364231 +0000 UTC m=+708.412276412"
Sep 15 21:20:00.800105 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:00.800032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:20:23.693480 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.693423    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"]
Sep 15 21:20:23.700752 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.700724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:23.702929 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.702906    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 21:20:23.703074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.702930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-96bbn\""
Sep 15 21:20:23.710123 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.710097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"]
Sep 15 21:20:23.785857 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.785816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6zrlv\" (UID: \"c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:23.785857 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.785863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvlq\" (UniqueName: \"kubernetes.io/projected/c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766-kube-api-access-mxvlq\") pod \"payload-pre-processing-59fbc47bf8-6zrlv\" (UID: \"c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:23.823497 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.823426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"]
Sep 15 21:20:23.827356 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.827318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:23.838067 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.838037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"]
Sep 15 21:20:23.886955 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.886906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/023c1a26-91e1-4d9a-8871-fe737cdac0c5-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fsqhb\" (UID: \"023c1a26-91e1-4d9a-8871-fe737cdac0c5\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:23.887150 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.886964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5m7\" (UniqueName: \"kubernetes.io/projected/023c1a26-91e1-4d9a-8871-fe737cdac0c5-kube-api-access-ts5m7\") pod \"payload-processing-5b54b7fbb7-fsqhb\" (UID: \"023c1a26-91e1-4d9a-8871-fe737cdac0c5\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:23.887150 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.887093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6zrlv\" (UID: \"c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:23.887150 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.887125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvlq\" (UniqueName: \"kubernetes.io/projected/c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766-kube-api-access-mxvlq\") pod \"payload-pre-processing-59fbc47bf8-6zrlv\" (UID: \"c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:23.887871 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.887853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6zrlv\" (UID: \"c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:23.896111 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.896070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvlq\" (UniqueName: \"kubernetes.io/projected/c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766-kube-api-access-mxvlq\") pod \"payload-pre-processing-59fbc47bf8-6zrlv\" (UID: \"c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:23.987920 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.987881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/023c1a26-91e1-4d9a-8871-fe737cdac0c5-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fsqhb\" (UID: \"023c1a26-91e1-4d9a-8871-fe737cdac0c5\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:23.987920 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.987921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5m7\" (UniqueName: \"kubernetes.io/projected/023c1a26-91e1-4d9a-8871-fe737cdac0c5-kube-api-access-ts5m7\") pod \"payload-processing-5b54b7fbb7-fsqhb\" (UID: \"023c1a26-91e1-4d9a-8871-fe737cdac0c5\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:23.988703 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.988676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/023c1a26-91e1-4d9a-8871-fe737cdac0c5-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fsqhb\" (UID: \"023c1a26-91e1-4d9a-8871-fe737cdac0c5\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:23.996245 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:23.996210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5m7\" (UniqueName: \"kubernetes.io/projected/023c1a26-91e1-4d9a-8871-fe737cdac0c5-kube-api-access-ts5m7\") pod \"payload-processing-5b54b7fbb7-fsqhb\" (UID: \"023c1a26-91e1-4d9a-8871-fe737cdac0c5\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:24.012961 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:24.012925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:24.139464 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:24.139422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:24.288570 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:24.288530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"]
Sep 15 21:20:24.291857 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:20:24.291822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80ba6c3_fd8e_4df1_bb8e_65ffcb47d766.slice/crio-e14cc395e68876f7bfc8e16a15742dadc437f3b877a9a99e77bf312b0a73cc98 WatchSource:0}: Error finding container e14cc395e68876f7bfc8e16a15742dadc437f3b877a9a99e77bf312b0a73cc98: Status 404 returned error can't find the container with id e14cc395e68876f7bfc8e16a15742dadc437f3b877a9a99e77bf312b0a73cc98
Sep 15 21:20:24.414235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:24.414211    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"]
Sep 15 21:20:24.417040 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:20:24.417009    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023c1a26_91e1_4d9a_8871_fe737cdac0c5.slice/crio-cd1be423c3b455e87eea8393eb8caf69650b5a66687e6b8197af1880e004a9ea WatchSource:0}: Error finding container cd1be423c3b455e87eea8393eb8caf69650b5a66687e6b8197af1880e004a9ea: Status 404 returned error can't find the container with id cd1be423c3b455e87eea8393eb8caf69650b5a66687e6b8197af1880e004a9ea
Sep 15 21:20:24.963707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:24.963667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv" event={"ID":"c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766","Type":"ContainerStarted","Data":"e14cc395e68876f7bfc8e16a15742dadc437f3b877a9a99e77bf312b0a73cc98"}
Sep 15 21:20:24.965149 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:24.965121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb" event={"ID":"023c1a26-91e1-4d9a-8871-fe737cdac0c5","Type":"ContainerStarted","Data":"cd1be423c3b455e87eea8393eb8caf69650b5a66687e6b8197af1880e004a9ea"}
Sep 15 21:20:25.971766 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:25.971733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb" event={"ID":"023c1a26-91e1-4d9a-8871-fe737cdac0c5","Type":"ContainerStarted","Data":"56f026200442b4700e0fa5c5e5e0d4dcf802d2925d2c0a30cfe7acb4906574a8"}
Sep 15 21:20:25.972257 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:25.971852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:25.973550 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:25.973520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv" event={"ID":"c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766","Type":"ContainerStarted","Data":"a6ece33a9180e93d56e67e781af66719eeb8931e42a0976a10e23ed36d8ef44f"}
Sep 15 21:20:25.973701 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:25.973660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:25.997839 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:25.997773    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb" podStartSLOduration=1.658211703 podStartE2EDuration="2.997751672s" totalImagesPullingTime="1.339539969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:25.99573206 +0000 UTC m=+744.595644173" watchObservedRunningTime="2026-09-15 21:20:25.997751672 +0000 UTC m=+744.597663782"
Sep 15 21:20:26.012732 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:26.012604    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv" podStartSLOduration=1.5452753860000001 podStartE2EDuration="3.012580003s" totalImagesPullingTime="1.467304617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:26.01172731 +0000 UTC m=+744.611639421" watchObservedRunningTime="2026-09-15 21:20:26.012580003 +0000 UTC m=+744.612492117"
Sep 15 21:20:36.980914 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:36.980881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6zrlv"
Sep 15 21:20:36.981331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:36.980953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-fsqhb"
Sep 15 21:20:59.926492 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:59.926436    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"]
Sep 15 21:20:59.931397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:59.931372    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:20:59.934021 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:59.933994    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 15 21:20:59.934196 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:59.934025    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2j7d9\""
Sep 15 21:20:59.934196 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:59.934117    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 21:20:59.934313 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:59.934239    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 21:20:59.940119 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:20:59.939878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"]
Sep 15 21:21:00.015278 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.015238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.015278 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.015277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1416c9-a759-4821-8323-4049dad5b202-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.015559 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.015378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.015559 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.015422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.015559 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.015471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.015559 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.015519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjx9b\" (UniqueName: \"kubernetes.io/projected/9e1416c9-a759-4821-8323-4049dad5b202-kube-api-access-qjx9b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.116503 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.116439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.116703 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.116540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.116703 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.116567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.116703 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.116604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjx9b\" (UniqueName: \"kubernetes.io/projected/9e1416c9-a759-4821-8323-4049dad5b202-kube-api-access-qjx9b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.116703 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.116691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.116928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.116718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1416c9-a759-4821-8323-4049dad5b202-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.116994 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.116922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.116994 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.116950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.117139 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.117114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.119328 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.119306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1416c9-a759-4821-8323-4049dad5b202-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.119488 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.119458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1416c9-a759-4821-8323-4049dad5b202-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.125687 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.125666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjx9b\" (UniqueName: \"kubernetes.io/projected/9e1416c9-a759-4821-8323-4049dad5b202-kube-api-access-qjx9b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-876k2\" (UID: \"9e1416c9-a759-4821-8323-4049dad5b202\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.243671 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.243628    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:00.520345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:00.520144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"]
Sep 15 21:21:00.522726 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:21:00.522695    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1416c9_a759_4821_8323_4049dad5b202.slice/crio-e37a74ea865732b15810cd37af687024aa6ef14356f5f18885c81612b507c539 WatchSource:0}: Error finding container e37a74ea865732b15810cd37af687024aa6ef14356f5f18885c81612b507c539: Status 404 returned error can't find the container with id e37a74ea865732b15810cd37af687024aa6ef14356f5f18885c81612b507c539
Sep 15 21:21:01.153776 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:01.153726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2" event={"ID":"9e1416c9-a759-4821-8323-4049dad5b202","Type":"ContainerStarted","Data":"e37a74ea865732b15810cd37af687024aa6ef14356f5f18885c81612b507c539"}
Sep 15 21:21:02.160852 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:02.160812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2" event={"ID":"9e1416c9-a759-4821-8323-4049dad5b202","Type":"ContainerStarted","Data":"ed8f3dabebc0fc3c46c1ed78a8cb693143f92546ed532295e9025871555a6782"}
Sep 15 21:21:02.179958 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:02.179899    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2" podStartSLOduration=2.171136398 podStartE2EDuration="3.179879483s" totalImagesPullingTime="1.008743085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:02.177414013 +0000 UTC m=+780.777326123" watchObservedRunningTime="2026-09-15 21:21:02.179879483 +0000 UTC m=+780.779791592"
Sep 15 21:21:10.244308 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:10.244268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:10.244772 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:10.244362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:10.257370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:10.257341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:11.219607 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:11.219576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-876k2"
Sep 15 21:21:37.423749 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.423713    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"]
Sep 15 21:21:37.428331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.428310    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.430607 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.430584    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 21:21:37.438527 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.438503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"]
Sep 15 21:21:37.442868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.442843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.443008 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.442872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.443008 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.442940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.443008 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.442969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp2qg\" (UniqueName: \"kubernetes.io/projected/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-kube-api-access-zp2qg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.443112 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.443010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.443112 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.443025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544151 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544151 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544408 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544408 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp2qg\" (UniqueName: \"kubernetes.io/projected/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-kube-api-access-zp2qg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544561 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544561 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544671 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544730 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.544789 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.544762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.546683 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.546658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.547150 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.547128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.554660 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.554639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp2qg\" (UniqueName: \"kubernetes.io/projected/e2c4caac-9cc7-4c73-b13d-a0003401e0a1-kube-api-access-zp2qg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t7wzf\" (UID: \"e2c4caac-9cc7-4c73-b13d-a0003401e0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:37.741937 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:37.741877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:38.018790 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:38.018737    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"]
Sep 15 21:21:38.021178 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:21:38.021147    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c4caac_9cc7_4c73_b13d_a0003401e0a1.slice/crio-58cd14e6edf08d33e014c37c92d46431c7f26dba40d444cf7b36e23a5ac50d50 WatchSource:0}: Error finding container 58cd14e6edf08d33e014c37c92d46431c7f26dba40d444cf7b36e23a5ac50d50: Status 404 returned error can't find the container with id 58cd14e6edf08d33e014c37c92d46431c7f26dba40d444cf7b36e23a5ac50d50
Sep 15 21:21:38.339022 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:38.338938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf" event={"ID":"e2c4caac-9cc7-4c73-b13d-a0003401e0a1","Type":"ContainerStarted","Data":"4b8a2c2d992a8423795dd147b1f66cd9038bf22211414a6264c747ca8350e552"}
Sep 15 21:21:38.339022 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:38.338994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf" event={"ID":"e2c4caac-9cc7-4c73-b13d-a0003401e0a1","Type":"ContainerStarted","Data":"58cd14e6edf08d33e014c37c92d46431c7f26dba40d444cf7b36e23a5ac50d50"}
Sep 15 21:21:38.357959 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:38.357889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf" podStartSLOduration=1.172664999 podStartE2EDuration="1.357873199s" totalImagesPullingTime="185.2082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:38.355899618 +0000 UTC m=+816.955811727" watchObservedRunningTime="2026-09-15 21:21:38.357873199 +0000 UTC m=+816.957785309"
Sep 15 21:21:47.742782 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:47.742744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:47.743302 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:47.742841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:47.755571 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:47.755544    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:21:48.409722 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:21:48.409687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t7wzf"
Sep 15 21:22:01.627692 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.627659    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"]
Sep 15 21:22:01.631365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.631338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.633507 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.633483    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 21:22:01.640785 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.640752    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"]
Sep 15 21:22:01.776636 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.776600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.776826 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.776643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcl6d\" (UniqueName: \"kubernetes.io/projected/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-kube-api-access-kcl6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.776826 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.776665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.776826 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.776797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.776955 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.776832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.776955 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.776866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.877654 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.877622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.877654 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.877660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.877967 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.877691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.877967 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.877732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.877967 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.877770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcl6d\" (UniqueName: \"kubernetes.io/projected/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-kube-api-access-kcl6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.877967 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.877804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.878179 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.878086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.878179 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.878147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.878375 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.878355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.880387 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.880367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 21:22:01.880387 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.880377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.887008 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.886985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcl6d\" (UniqueName: \"kubernetes.io/projected/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-kube-api-access-kcl6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.891420 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.891398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5e7642-1bcc-41b5-9c17-c30fe08e026f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcx9v\" (UID: \"1e5e7642-1bcc-41b5-9c17-c30fe08e026f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:01.942322 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:01.942294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:02.221930 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:02.221903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"]
Sep 15 21:22:02.224589 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:22:02.224559    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5e7642_1bcc_41b5_9c17_c30fe08e026f.slice/crio-1b0e637fe0dc6ae44a8d8005cd8458d6c070f7758a15d6d4acbe745a2cf8a2d3 WatchSource:0}: Error finding container 1b0e637fe0dc6ae44a8d8005cd8458d6c070f7758a15d6d4acbe745a2cf8a2d3: Status 404 returned error can't find the container with id 1b0e637fe0dc6ae44a8d8005cd8458d6c070f7758a15d6d4acbe745a2cf8a2d3
Sep 15 21:22:02.473854 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:02.473789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v" event={"ID":"1e5e7642-1bcc-41b5-9c17-c30fe08e026f","Type":"ContainerStarted","Data":"1b0e637fe0dc6ae44a8d8005cd8458d6c070f7758a15d6d4acbe745a2cf8a2d3"}
Sep 15 21:22:03.480504 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:03.480469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v" event={"ID":"1e5e7642-1bcc-41b5-9c17-c30fe08e026f","Type":"ContainerStarted","Data":"d09d2eeb3f2cf604d6f090177dbe9df3bcd726a7fa31b5e95d9eb1ea9ea4b7ad"}
Sep 15 21:22:03.501912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:03.501854    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v" podStartSLOduration=2.29538231 podStartE2EDuration="2.501836206s" totalImagesPullingTime="206.453896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:03.500694392 +0000 UTC m=+842.100606504" watchObservedRunningTime="2026-09-15 21:22:03.501836206 +0000 UTC m=+842.101748316"
Sep 15 21:22:11.943355 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:11.943314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:11.943862 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:11.943388    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:11.956265 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:11.956233    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:22:12.540043 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:22:12.540011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcx9v"
Sep 15 21:24:00.391959 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.391921    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"]
Sep 15 21:24:00.396161 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.396133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.399032 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.398992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 21:24:00.399032 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.399015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vzmv2\""
Sep 15 21:24:00.399032 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.399024    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:24:00.399366 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.399348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-625609-serving-cert\""
Sep 15 21:24:00.399366 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.399358    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 21:24:00.405463 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.405368    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"]
Sep 15 21:24:00.483991 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.483952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c753f4-7f52-4867-8345-6a0efef1be86-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.484182 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.484018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbnq\" (UniqueName: \"kubernetes.io/projected/a4c753f4-7f52-4867-8345-6a0efef1be86-kube-api-access-4vbnq\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.484182 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.484124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-metrics-certs\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.484182 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.484163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.585687 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.585647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c753f4-7f52-4867-8345-6a0efef1be86-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.585868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.585704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbnq\" (UniqueName: \"kubernetes.io/projected/a4c753f4-7f52-4867-8345-6a0efef1be86-kube-api-access-4vbnq\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.585868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.585754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-metrics-certs\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.585868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.585777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.585975 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:24:00.585888    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-serving-cert: secret "maas-api-e2e-worker-w4-625609-serving-cert" not found
Sep 15 21:24:00.585975 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:24:00.585961    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls podName:a4c753f4-7f52-4867-8345-6a0efef1be86 nodeName:}" failed. No retries permitted until 2026-09-15 21:24:01.085941544 +0000 UTC m=+959.685853633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls") pod "maas-api-e2e-worker-w4-625609-956b54778-zz69m" (UID: "a4c753f4-7f52-4867-8345-6a0efef1be86") : secret "maas-api-e2e-worker-w4-625609-serving-cert" not found
Sep 15 21:24:00.586331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.586302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c753f4-7f52-4867-8345-6a0efef1be86-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.588365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.588347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-metrics-certs\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:00.593847 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:00.593823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbnq\" (UniqueName: \"kubernetes.io/projected/a4c753f4-7f52-4867-8345-6a0efef1be86-kube-api-access-4vbnq\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:01.091603 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.091568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:01.094213 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.094180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls\") pod \"maas-api-e2e-worker-w4-625609-956b54778-zz69m\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:01.100264 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.100233    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"]
Sep 15 21:24:01.104744 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.104722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:01.112081 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.112055    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-625609\""
Sep 15 21:24:01.115668 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.115648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-625609-dockercfg-8hnzl\""
Sep 15 21:24:01.134005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.133963    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"]
Sep 15 21:24:01.193019 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.192975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a176feb7-0805-4413-a305-07fe3130200a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z\" (UID: \"a176feb7-0805-4413-a305-07fe3130200a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:01.193211 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.193118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pm2c\" (UniqueName: \"kubernetes.io/projected/a176feb7-0805-4413-a305-07fe3130200a-kube-api-access-6pm2c\") pod \"payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z\" (UID: \"a176feb7-0805-4413-a305-07fe3130200a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:01.294302 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.294250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pm2c\" (UniqueName: \"kubernetes.io/projected/a176feb7-0805-4413-a305-07fe3130200a-kube-api-access-6pm2c\") pod \"payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z\" (UID: \"a176feb7-0805-4413-a305-07fe3130200a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:01.294520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.294385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a176feb7-0805-4413-a305-07fe3130200a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z\" (UID: \"a176feb7-0805-4413-a305-07fe3130200a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:01.295527 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.295501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a176feb7-0805-4413-a305-07fe3130200a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z\" (UID: \"a176feb7-0805-4413-a305-07fe3130200a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:01.304359 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.304326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pm2c\" (UniqueName: \"kubernetes.io/projected/a176feb7-0805-4413-a305-07fe3130200a-kube-api-access-6pm2c\") pod \"payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z\" (UID: \"a176feb7-0805-4413-a305-07fe3130200a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:01.311671 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.311639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:01.416478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.416428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:01.605563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.605216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"]
Sep 15 21:24:01.611732 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.611707    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:24:01.697184 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:01.697158    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"]
Sep 15 21:24:01.699616 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:01.699585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda176feb7_0805_4413_a305_07fe3130200a.slice/crio-1bdc90de0c8ea0b4e96562e185c49ac20c6a6c7fafd9a0ff8350996d42ab9f19 WatchSource:0}: Error finding container 1bdc90de0c8ea0b4e96562e185c49ac20c6a6c7fafd9a0ff8350996d42ab9f19: Status 404 returned error can't find the container with id 1bdc90de0c8ea0b4e96562e185c49ac20c6a6c7fafd9a0ff8350996d42ab9f19
Sep 15 21:24:02.099971 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.099921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z" event={"ID":"a176feb7-0805-4413-a305-07fe3130200a","Type":"ContainerStarted","Data":"9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62"}
Sep 15 21:24:02.100176 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.099991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z" event={"ID":"a176feb7-0805-4413-a305-07fe3130200a","Type":"ContainerStarted","Data":"1bdc90de0c8ea0b4e96562e185c49ac20c6a6c7fafd9a0ff8350996d42ab9f19"}
Sep 15 21:24:02.100176 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.100103    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:02.101613 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.101590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m" event={"ID":"a4c753f4-7f52-4867-8345-6a0efef1be86","Type":"ContainerStarted","Data":"9552c435489f7ff664e265e0dfa3ef5925e6e5603925e49e1fcd259f6e338650"}
Sep 15 21:24:02.123271 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.123215    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z" podStartSLOduration=1.123193873 podStartE2EDuration="1.123193873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:02.118565003 +0000 UTC m=+960.718477147" watchObservedRunningTime="2026-09-15 21:24:02.123193873 +0000 UTC m=+960.723105983"
Sep 15 21:24:02.510065 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.509878    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"]
Sep 15 21:24:02.514178 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.514152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:02.518077 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.516735    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a276f9-dockercfg-78wzk\""
Sep 15 21:24:02.518077 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.516783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a276f9\""
Sep 15 21:24:02.524222 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.524196    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"]
Sep 15 21:24:02.607419 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.607377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7727f857-7018-4cd0-ac20-6ea0917cf48c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr\" (UID: \"7727f857-7018-4cd0-ac20-6ea0917cf48c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:02.607592 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.607538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df5br\" (UniqueName: \"kubernetes.io/projected/7727f857-7018-4cd0-ac20-6ea0917cf48c-kube-api-access-df5br\") pod \"payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr\" (UID: \"7727f857-7018-4cd0-ac20-6ea0917cf48c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:02.708706 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.708664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7727f857-7018-4cd0-ac20-6ea0917cf48c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr\" (UID: \"7727f857-7018-4cd0-ac20-6ea0917cf48c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:02.708895 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.708757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df5br\" (UniqueName: \"kubernetes.io/projected/7727f857-7018-4cd0-ac20-6ea0917cf48c-kube-api-access-df5br\") pod \"payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr\" (UID: \"7727f857-7018-4cd0-ac20-6ea0917cf48c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:02.709631 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.709602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7727f857-7018-4cd0-ac20-6ea0917cf48c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr\" (UID: \"7727f857-7018-4cd0-ac20-6ea0917cf48c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:02.721355 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.721323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df5br\" (UniqueName: \"kubernetes.io/projected/7727f857-7018-4cd0-ac20-6ea0917cf48c-kube-api-access-df5br\") pod \"payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr\" (UID: \"7727f857-7018-4cd0-ac20-6ea0917cf48c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:02.831775 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:02.831696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:03.128605 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.128571    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"]
Sep 15 21:24:03.628800 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.628766    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"]
Sep 15 21:24:03.638476 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.638425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:03.641507 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.641002    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-70880c\""
Sep 15 21:24:03.641507 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.641181    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-70880c-dockercfg-qjd9z\""
Sep 15 21:24:03.645127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.645052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"]
Sep 15 21:24:03.720430 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.720388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca1d149-6629-469a-a3ac-ed7e35438b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb\" (UID: \"9ca1d149-6629-469a-a3ac-ed7e35438b85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:03.720647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.720496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4krk\" (UniqueName: \"kubernetes.io/projected/9ca1d149-6629-469a-a3ac-ed7e35438b85-kube-api-access-q4krk\") pod \"payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb\" (UID: \"9ca1d149-6629-469a-a3ac-ed7e35438b85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:03.826174 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.824684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca1d149-6629-469a-a3ac-ed7e35438b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb\" (UID: \"9ca1d149-6629-469a-a3ac-ed7e35438b85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:03.826174 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.824750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4krk\" (UniqueName: \"kubernetes.io/projected/9ca1d149-6629-469a-a3ac-ed7e35438b85-kube-api-access-q4krk\") pod \"payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb\" (UID: \"9ca1d149-6629-469a-a3ac-ed7e35438b85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:03.826174 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.825958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca1d149-6629-469a-a3ac-ed7e35438b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb\" (UID: \"9ca1d149-6629-469a-a3ac-ed7e35438b85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:03.836685 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.836655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4krk\" (UniqueName: \"kubernetes.io/projected/9ca1d149-6629-469a-a3ac-ed7e35438b85-kube-api-access-q4krk\") pod \"payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb\" (UID: \"9ca1d149-6629-469a-a3ac-ed7e35438b85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:03.953815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:03.953663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:04.318129 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:04.318090    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7727f857_7018_4cd0_ac20_6ea0917cf48c.slice/crio-031b2d5d993e10fc1255e35951d0cc3e8178ecddf62b22cd9de2cee2d694dbdb WatchSource:0}: Error finding container 031b2d5d993e10fc1255e35951d0cc3e8178ecddf62b22cd9de2cee2d694dbdb: Status 404 returned error can't find the container with id 031b2d5d993e10fc1255e35951d0cc3e8178ecddf62b22cd9de2cee2d694dbdb
Sep 15 21:24:04.594696 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:04.594627    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"]
Sep 15 21:24:04.597557 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:04.597516    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca1d149_6629_469a_a3ac_ed7e35438b85.slice/crio-2266e12e8b9f7daadb121ac13f9166c7a6878e4f7a5b7c51a925755e9c10f623 WatchSource:0}: Error finding container 2266e12e8b9f7daadb121ac13f9166c7a6878e4f7a5b7c51a925755e9c10f623: Status 404 returned error can't find the container with id 2266e12e8b9f7daadb121ac13f9166c7a6878e4f7a5b7c51a925755e9c10f623
Sep 15 21:24:05.121839 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.121781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m" event={"ID":"a4c753f4-7f52-4867-8345-6a0efef1be86","Type":"ContainerStarted","Data":"d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514"}
Sep 15 21:24:05.122332 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.122081    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:05.123789 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.123760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr" event={"ID":"7727f857-7018-4cd0-ac20-6ea0917cf48c","Type":"ContainerStarted","Data":"7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857"}
Sep 15 21:24:05.123983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.123814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr" event={"ID":"7727f857-7018-4cd0-ac20-6ea0917cf48c","Type":"ContainerStarted","Data":"031b2d5d993e10fc1255e35951d0cc3e8178ecddf62b22cd9de2cee2d694dbdb"}
Sep 15 21:24:05.123983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.123847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:05.125413 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.125390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb" event={"ID":"9ca1d149-6629-469a-a3ac-ed7e35438b85","Type":"ContainerStarted","Data":"75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3"}
Sep 15 21:24:05.125578 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.125435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb" event={"ID":"9ca1d149-6629-469a-a3ac-ed7e35438b85","Type":"ContainerStarted","Data":"2266e12e8b9f7daadb121ac13f9166c7a6878e4f7a5b7c51a925755e9c10f623"}
Sep 15 21:24:05.125578 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.125489    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:05.147334 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.147274    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m" podStartSLOduration=2.364085103 podStartE2EDuration="5.147256994s" totalImagesPullingTime="2.783171891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:05.144651911 +0000 UTC m=+963.744564011" watchObservedRunningTime="2026-09-15 21:24:05.147256994 +0000 UTC m=+963.747169108"
Sep 15 21:24:05.167224 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.167175    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb" podStartSLOduration=2.167162703 podStartE2EDuration="2.167162703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:05.165354325 +0000 UTC m=+963.765266434" watchObservedRunningTime="2026-09-15 21:24:05.167162703 +0000 UTC m=+963.767074813"
Sep 15 21:24:05.185595 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:05.185521    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr" podStartSLOduration=3.185504234 podStartE2EDuration="3.185504234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:05.184820537 +0000 UTC m=+963.784732646" watchObservedRunningTime="2026-09-15 21:24:05.185504234 +0000 UTC m=+963.785416345"
Sep 15 21:24:11.136995 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:11.136959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:24:12.264882 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.264845    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"]
Sep 15 21:24:12.269343 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.269325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:12.273028 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.273011    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-44fbc30a\""
Sep 15 21:24:12.273270 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.273257    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-44fbc30a-dockercfg-nwjdz\""
Sep 15 21:24:12.279749 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.279726    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"]
Sep 15 21:24:12.327900 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.327852    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"]
Sep 15 21:24:12.332178 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.332158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:12.343423 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.343398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"]
Sep 15 21:24:12.408426 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.408388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nz5w\" (UniqueName: \"kubernetes.io/projected/05f7e17c-98f6-4d47-8af5-4e457fde209e-kube-api-access-5nz5w\") pod \"payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc\" (UID: \"05f7e17c-98f6-4d47-8af5-4e457fde209e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:12.408603 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.408465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f7e17c-98f6-4d47-8af5-4e457fde209e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc\" (UID: \"05f7e17c-98f6-4d47-8af5-4e457fde209e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:12.510071 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.510027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nz5w\" (UniqueName: \"kubernetes.io/projected/05f7e17c-98f6-4d47-8af5-4e457fde209e-kube-api-access-5nz5w\") pod \"payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc\" (UID: \"05f7e17c-98f6-4d47-8af5-4e457fde209e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:12.510269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.510086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-plugins-config-volume\") pod \"payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7\" (UID: \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\") " pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:12.510269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.510111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjjp\" (UniqueName: \"kubernetes.io/projected/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-kube-api-access-jpjjp\") pod \"payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7\" (UID: \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\") " pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:12.510269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.510126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f7e17c-98f6-4d47-8af5-4e457fde209e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc\" (UID: \"05f7e17c-98f6-4d47-8af5-4e457fde209e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:12.510925 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.510900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f7e17c-98f6-4d47-8af5-4e457fde209e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc\" (UID: \"05f7e17c-98f6-4d47-8af5-4e457fde209e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:12.518911 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.518851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nz5w\" (UniqueName: \"kubernetes.io/projected/05f7e17c-98f6-4d47-8af5-4e457fde209e-kube-api-access-5nz5w\") pod \"payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc\" (UID: \"05f7e17c-98f6-4d47-8af5-4e457fde209e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:12.581393 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.581353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:12.611367 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.611336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-plugins-config-volume\") pod \"payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7\" (UID: \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\") " pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:12.611566 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.611377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjjp\" (UniqueName: \"kubernetes.io/projected/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-kube-api-access-jpjjp\") pod \"payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7\" (UID: \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\") " pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:12.612154 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.612133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-plugins-config-volume\") pod \"payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7\" (UID: \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\") " pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:12.621687 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.621626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjjp\" (UniqueName: \"kubernetes.io/projected/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-kube-api-access-jpjjp\") pod \"payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7\" (UID: \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\") " pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:12.645376 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.644591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:12.860653 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.860629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"]
Sep 15 21:24:12.862652 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:12.862628    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f7e17c_98f6_4d47_8af5_4e457fde209e.slice/crio-e9753cc8b0bf761752c6ba04dee2f22a43f1638d62b9cd64d11f4ffc62f477a1 WatchSource:0}: Error finding container e9753cc8b0bf761752c6ba04dee2f22a43f1638d62b9cd64d11f4ffc62f477a1: Status 404 returned error can't find the container with id e9753cc8b0bf761752c6ba04dee2f22a43f1638d62b9cd64d11f4ffc62f477a1
Sep 15 21:24:12.949869 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:12.949841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"]
Sep 15 21:24:12.952030 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:12.952000    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbe4cc1_3536_4c5d_8032_c2ed53ecd183.slice/crio-8813b0a0140b904dec204a6cf59e12e2f81557655cb0142215f6b594bbe831ae WatchSource:0}: Error finding container 8813b0a0140b904dec204a6cf59e12e2f81557655cb0142215f6b594bbe831ae: Status 404 returned error can't find the container with id 8813b0a0140b904dec204a6cf59e12e2f81557655cb0142215f6b594bbe831ae
Sep 15 21:24:13.109392 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.109361    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:24:13.173754 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.173651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc" event={"ID":"05f7e17c-98f6-4d47-8af5-4e457fde209e","Type":"ContainerStarted","Data":"c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e"}
Sep 15 21:24:13.173754 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.173715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc" event={"ID":"05f7e17c-98f6-4d47-8af5-4e457fde209e","Type":"ContainerStarted","Data":"e9753cc8b0bf761752c6ba04dee2f22a43f1638d62b9cd64d11f4ffc62f477a1"}
Sep 15 21:24:13.173999 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.173816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:13.175762 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.175720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7" event={"ID":"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183","Type":"ContainerStarted","Data":"d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c"}
Sep 15 21:24:13.175877 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.175773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7" event={"ID":"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183","Type":"ContainerStarted","Data":"8813b0a0140b904dec204a6cf59e12e2f81557655cb0142215f6b594bbe831ae"}
Sep 15 21:24:13.176249 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.176233    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:13.196935 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.196743    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc" podStartSLOduration=1.196717592 podStartE2EDuration="1.196717592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:13.193586083 +0000 UTC m=+971.793498207" watchObservedRunningTime="2026-09-15 21:24:13.196717592 +0000 UTC m=+971.796629702"
Sep 15 21:24:13.215600 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.215364    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7" podStartSLOduration=1.215340606 podStartE2EDuration="1.215340606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:13.21228441 +0000 UTC m=+971.812196523" watchObservedRunningTime="2026-09-15 21:24:13.215340606 +0000 UTC m=+971.815252717"
Sep 15 21:24:13.392623 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.392585    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"]
Sep 15 21:24:13.396711 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.396689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:13.399362 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.399334    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-faa4c5\""
Sep 15 21:24:13.430926 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.430842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"]
Sep 15 21:24:13.522127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.522090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2196d395-2660-4ec0-91d9-05331f52e144-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5\" (UID: \"2196d395-2660-4ec0-91d9-05331f52e144\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:13.522520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.522491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgcv\" (UniqueName: \"kubernetes.io/projected/2196d395-2660-4ec0-91d9-05331f52e144-kube-api-access-bqgcv\") pod \"payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5\" (UID: \"2196d395-2660-4ec0-91d9-05331f52e144\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:13.623683 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.623636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgcv\" (UniqueName: \"kubernetes.io/projected/2196d395-2660-4ec0-91d9-05331f52e144-kube-api-access-bqgcv\") pod \"payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5\" (UID: \"2196d395-2660-4ec0-91d9-05331f52e144\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:13.623856 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.623809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2196d395-2660-4ec0-91d9-05331f52e144-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5\" (UID: \"2196d395-2660-4ec0-91d9-05331f52e144\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:13.624569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.624550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2196d395-2660-4ec0-91d9-05331f52e144-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5\" (UID: \"2196d395-2660-4ec0-91d9-05331f52e144\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:13.639955 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.639932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgcv\" (UniqueName: \"kubernetes.io/projected/2196d395-2660-4ec0-91d9-05331f52e144-kube-api-access-bqgcv\") pod \"payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5\" (UID: \"2196d395-2660-4ec0-91d9-05331f52e144\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:13.708714 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.708679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:13.998040 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:13.998013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"]
Sep 15 21:24:14.000092 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:14.000047    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2196d395_2660_4ec0_91d9_05331f52e144.slice/crio-7f608602c004096b9ce64cfe22d2bde8db7fe08884d0292fcbce0950f88174be WatchSource:0}: Error finding container 7f608602c004096b9ce64cfe22d2bde8db7fe08884d0292fcbce0950f88174be: Status 404 returned error can't find the container with id 7f608602c004096b9ce64cfe22d2bde8db7fe08884d0292fcbce0950f88174be
Sep 15 21:24:14.184017 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:14.183982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5" event={"ID":"2196d395-2660-4ec0-91d9-05331f52e144","Type":"ContainerStarted","Data":"8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315"}
Sep 15 21:24:14.184208 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:14.184039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5" event={"ID":"2196d395-2660-4ec0-91d9-05331f52e144","Type":"ContainerStarted","Data":"7f608602c004096b9ce64cfe22d2bde8db7fe08884d0292fcbce0950f88174be"}
Sep 15 21:24:14.184380 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:14.184362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:14.204648 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:14.204591    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5" podStartSLOduration=1.204572275 podStartE2EDuration="1.204572275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:14.20451241 +0000 UTC m=+972.804424520" watchObservedRunningTime="2026-09-15 21:24:14.204572275 +0000 UTC m=+972.804484389"
Sep 15 21:24:16.133880 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:16.133842    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:24:16.134347 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:16.134120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:24:24.186316 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:24.186218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:25.191231 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:25.191200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:24:25.191736 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:25.191491    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:28.632233 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:28.632175    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"]
Sep 15 21:24:28.640663 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:28.639611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"
Sep 15 21:24:28.643380 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:28.643105    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7fhv\""
Sep 15 21:24:28.649735 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:28.649681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"]
Sep 15 21:24:28.659706 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:28.659660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2781e8-1e0b-4ca1-99a7-19db81bd195c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf\" (UID: \"6f2781e8-1e0b-4ca1-99a7-19db81bd195c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"
Sep 15 21:24:28.760277 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:28.760228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2781e8-1e0b-4ca1-99a7-19db81bd195c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf\" (UID: \"6f2781e8-1e0b-4ca1-99a7-19db81bd195c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"
Sep 15 21:24:28.761838 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:28.761805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2781e8-1e0b-4ca1-99a7-19db81bd195c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf\" (UID: \"6f2781e8-1e0b-4ca1-99a7-19db81bd195c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"
Sep 15 21:24:28.959246 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:28.959208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"
Sep 15 21:24:29.281199 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:29.281121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"]
Sep 15 21:24:29.285343 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:29.285310    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2781e8_1e0b_4ca1_99a7_19db81bd195c.slice/crio-650fed8478ea2eddc02ad39050ece5bad585ff99ed8c0e11ce67fa0322a32655 WatchSource:0}: Error finding container 650fed8478ea2eddc02ad39050ece5bad585ff99ed8c0e11ce67fa0322a32655: Status 404 returned error can't find the container with id 650fed8478ea2eddc02ad39050ece5bad585ff99ed8c0e11ce67fa0322a32655
Sep 15 21:24:30.287360 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:30.287318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf" event={"ID":"6f2781e8-1e0b-4ca1-99a7-19db81bd195c","Type":"ContainerStarted","Data":"650fed8478ea2eddc02ad39050ece5bad585ff99ed8c0e11ce67fa0322a32655"}
Sep 15 21:24:32.302749 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.302715    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f2781e8-1e0b-4ca1-99a7-19db81bd195c" containerID="b09d953202e6d776e20e9342f20b3d7034155168cdee1a0014292bb02b5d9c2e" exitCode=0
Sep 15 21:24:32.303178 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.302777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf" event={"ID":"6f2781e8-1e0b-4ca1-99a7-19db81bd195c","Type":"ContainerDied","Data":"b09d953202e6d776e20e9342f20b3d7034155168cdee1a0014292bb02b5d9c2e"}
Sep 15 21:24:32.747375 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.747341    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"]
Sep 15 21:24:32.751388 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.751363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.753883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.753860    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-779ace-serving-cert\""
Sep 15 21:24:32.767064 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.767034    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"]
Sep 15 21:24:32.807177 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.807138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lg7\" (UniqueName: \"kubernetes.io/projected/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-kube-api-access-67lg7\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.807373 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.807201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-openshift-service-ca\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.807373 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.807229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-metrics-certs\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.807373 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.807291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.908690 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.908651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.908892 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.908747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lg7\" (UniqueName: \"kubernetes.io/projected/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-kube-api-access-67lg7\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.908892 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.908797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-openshift-service-ca\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.908892 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:24:32.908816    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-serving-cert: secret "maas-api-e2e-inf-b-779ace-serving-cert" not found
Sep 15 21:24:32.908892 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.908820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-metrics-certs\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.908892 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:24:32.908882    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls podName:bf1d0a7b-7e3c-4fb2-ace6-a180dff25954 nodeName:}" failed. No retries permitted until 2026-09-15 21:24:33.408860432 +0000 UTC m=+992.008772543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls") pod "maas-api-e2e-inf-b-779ace-768dc97756-wxhh9" (UID: "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954") : secret "maas-api-e2e-inf-b-779ace-serving-cert" not found
Sep 15 21:24:32.909522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.909500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-openshift-service-ca\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.911617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.911597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-metrics-certs\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:32.917899 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:32.917878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lg7\" (UniqueName: \"kubernetes.io/projected/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-kube-api-access-67lg7\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:33.414187 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.414156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:33.416935 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.416897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls\") pod \"maas-api-e2e-inf-b-779ace-768dc97756-wxhh9\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:33.467238 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.467209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"
Sep 15 21:24:33.515184 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.515138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2781e8-1e0b-4ca1-99a7-19db81bd195c-maas-api-service-ca\") pod \"6f2781e8-1e0b-4ca1-99a7-19db81bd195c\" (UID: \"6f2781e8-1e0b-4ca1-99a7-19db81bd195c\") "
Sep 15 21:24:33.515596 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.515568    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2781e8-1e0b-4ca1-99a7-19db81bd195c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f2781e8-1e0b-4ca1-99a7-19db81bd195c" (UID: "6f2781e8-1e0b-4ca1-99a7-19db81bd195c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:24:33.617124 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.617017    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2781e8-1e0b-4ca1-99a7-19db81bd195c-maas-api-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:24:33.637566 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.637533    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"]
Sep 15 21:24:33.638063 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.638049    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2781e8-1e0b-4ca1-99a7-19db81bd195c" containerName="revoke-keys"
Sep 15 21:24:33.638123 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.638065    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2781e8-1e0b-4ca1-99a7-19db81bd195c" containerName="revoke-keys"
Sep 15 21:24:33.638160 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.638127    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2781e8-1e0b-4ca1-99a7-19db81bd195c" containerName="revoke-keys"
Sep 15 21:24:33.641487 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.641459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:33.648583 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.648434    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-779ace\""
Sep 15 21:24:33.648852 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.648832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-779ace-dockercfg-7rpw7\""
Sep 15 21:24:33.665891 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.665842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:33.667210 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.667184    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"]
Sep 15 21:24:33.718478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.718419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1d33a5-8be5-46b7-b59e-166055295498-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25\" (UID: \"ff1d33a5-8be5-46b7-b59e-166055295498\") " pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:33.718701 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.718667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg64w\" (UniqueName: \"kubernetes.io/projected/ff1d33a5-8be5-46b7-b59e-166055295498-kube-api-access-hg64w\") pod \"payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25\" (UID: \"ff1d33a5-8be5-46b7-b59e-166055295498\") " pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:33.819899 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.819870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg64w\" (UniqueName: \"kubernetes.io/projected/ff1d33a5-8be5-46b7-b59e-166055295498-kube-api-access-hg64w\") pod \"payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25\" (UID: \"ff1d33a5-8be5-46b7-b59e-166055295498\") " pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:33.820063 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.819918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1d33a5-8be5-46b7-b59e-166055295498-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25\" (UID: \"ff1d33a5-8be5-46b7-b59e-166055295498\") " pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:33.820646 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.820627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1d33a5-8be5-46b7-b59e-166055295498-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25\" (UID: \"ff1d33a5-8be5-46b7-b59e-166055295498\") " pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:33.828634 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.828605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg64w\" (UniqueName: \"kubernetes.io/projected/ff1d33a5-8be5-46b7-b59e-166055295498-kube-api-access-hg64w\") pod \"payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25\" (UID: \"ff1d33a5-8be5-46b7-b59e-166055295498\") " pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:33.952563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.952529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:33.955382 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:33.955362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"]
Sep 15 21:24:33.957964 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:33.957937    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1d0a7b_7e3c_4fb2_ace6_a180dff25954.slice/crio-b751686040ae75320a365c806e80cd48f9b1f28832a8cf2c19413aff07bebaab WatchSource:0}: Error finding container b751686040ae75320a365c806e80cd48f9b1f28832a8cf2c19413aff07bebaab: Status 404 returned error can't find the container with id b751686040ae75320a365c806e80cd48f9b1f28832a8cf2c19413aff07bebaab
Sep 15 21:24:34.242744 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:34.242712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"]
Sep 15 21:24:34.246033 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:34.246004    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1d33a5_8be5_46b7_b59e_166055295498.slice/crio-a9d481098e056004fbb06082895220709213c83ba795af2f028a08bd2e78945c WatchSource:0}: Error finding container a9d481098e056004fbb06082895220709213c83ba795af2f028a08bd2e78945c: Status 404 returned error can't find the container with id a9d481098e056004fbb06082895220709213c83ba795af2f028a08bd2e78945c
Sep 15 21:24:34.316040 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:34.315973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25" event={"ID":"ff1d33a5-8be5-46b7-b59e-166055295498","Type":"ContainerStarted","Data":"a9d481098e056004fbb06082895220709213c83ba795af2f028a08bd2e78945c"}
Sep 15 21:24:34.317510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:34.317488    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"
Sep 15 21:24:34.317671 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:34.317490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf" event={"ID":"6f2781e8-1e0b-4ca1-99a7-19db81bd195c","Type":"ContainerDied","Data":"650fed8478ea2eddc02ad39050ece5bad585ff99ed8c0e11ce67fa0322a32655"}
Sep 15 21:24:34.317671 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:34.317612    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650fed8478ea2eddc02ad39050ece5bad585ff99ed8c0e11ce67fa0322a32655"
Sep 15 21:24:34.319049 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:34.319012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9" event={"ID":"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954","Type":"ContainerStarted","Data":"b751686040ae75320a365c806e80cd48f9b1f28832a8cf2c19413aff07bebaab"}
Sep 15 21:24:35.328160 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:35.328114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9" event={"ID":"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954","Type":"ContainerStarted","Data":"e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289"}
Sep 15 21:24:35.328667 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:35.328202    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:35.329959 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:35.329930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25" event={"ID":"ff1d33a5-8be5-46b7-b59e-166055295498","Type":"ContainerStarted","Data":"ea5b377b0ec5b22599e79f4c5a722939840460e10e182bb22c089ecbe44b0c3d"}
Sep 15 21:24:35.330084 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:35.330053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:35.361425 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:35.361365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9" podStartSLOduration=2.9411291520000002 podStartE2EDuration="3.361345273s" totalImagesPullingTime="420.216121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:35.360607173 +0000 UTC m=+993.960519285" watchObservedRunningTime="2026-09-15 21:24:35.361345273 +0000 UTC m=+993.961257394"
Sep 15 21:24:35.390947 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:35.390886    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25" podStartSLOduration=2.390863564 podStartE2EDuration="2.390863564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:35.388847018 +0000 UTC m=+993.988759139" watchObservedRunningTime="2026-09-15 21:24:35.390863564 +0000 UTC m=+993.990775674"
Sep 15 21:24:38.753346 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:38.753305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"]
Sep 15 21:24:38.753835 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:38.753594    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7" podUID="ffbe4cc1-3536-4c5d-8032-c2ed53ecd183" containerName="payload-processing" containerID="cri-o://d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c" gracePeriod=30
Sep 15 21:24:38.769187 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:38.769154    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"]
Sep 15 21:24:38.769425 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:38.769399    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc" podUID="05f7e17c-98f6-4d47-8af5-4e457fde209e" containerName="payload-pre-processing" containerID="cri-o://c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e" gracePeriod=30
Sep 15 21:24:39.065560 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.065533    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:39.067581 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.067555    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:39.071308 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.071282    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"]
Sep 15 21:24:39.082597 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.082570    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fbc30a-b98pf"]
Sep 15 21:24:39.172800 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.172768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f7e17c-98f6-4d47-8af5-4e457fde209e-plugins-config-volume\") pod \"05f7e17c-98f6-4d47-8af5-4e457fde209e\" (UID: \"05f7e17c-98f6-4d47-8af5-4e457fde209e\") "
Sep 15 21:24:39.172986 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.172813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-plugins-config-volume\") pod \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\" (UID: \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\") "
Sep 15 21:24:39.172986 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.172965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nz5w\" (UniqueName: \"kubernetes.io/projected/05f7e17c-98f6-4d47-8af5-4e457fde209e-kube-api-access-5nz5w\") pod \"05f7e17c-98f6-4d47-8af5-4e457fde209e\" (UID: \"05f7e17c-98f6-4d47-8af5-4e457fde209e\") "
Sep 15 21:24:39.173108 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.172998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpjjp\" (UniqueName: \"kubernetes.io/projected/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-kube-api-access-jpjjp\") pod \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\" (UID: \"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183\") "
Sep 15 21:24:39.173246 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.173214    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f7e17c-98f6-4d47-8af5-4e457fde209e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05f7e17c-98f6-4d47-8af5-4e457fde209e" (UID: "05f7e17c-98f6-4d47-8af5-4e457fde209e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:24:39.173338 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.173281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffbe4cc1-3536-4c5d-8032-c2ed53ecd183" (UID: "ffbe4cc1-3536-4c5d-8032-c2ed53ecd183"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:24:39.173553 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.173526    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f7e17c-98f6-4d47-8af5-4e457fde209e-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:24:39.173668 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.173554    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:24:39.175795 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.175767    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-kube-api-access-jpjjp" (OuterVolumeSpecName: "kube-api-access-jpjjp") pod "ffbe4cc1-3536-4c5d-8032-c2ed53ecd183" (UID: "ffbe4cc1-3536-4c5d-8032-c2ed53ecd183"). InnerVolumeSpecName "kube-api-access-jpjjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:39.175910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.175805    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f7e17c-98f6-4d47-8af5-4e457fde209e-kube-api-access-5nz5w" (OuterVolumeSpecName: "kube-api-access-5nz5w") pod "05f7e17c-98f6-4d47-8af5-4e457fde209e" (UID: "05f7e17c-98f6-4d47-8af5-4e457fde209e"). InnerVolumeSpecName "kube-api-access-5nz5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:39.274317 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.274270    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nz5w\" (UniqueName: \"kubernetes.io/projected/05f7e17c-98f6-4d47-8af5-4e457fde209e-kube-api-access-5nz5w\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:24:39.274317 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.274308    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjjp\" (UniqueName: \"kubernetes.io/projected/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183-kube-api-access-jpjjp\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:24:39.356370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.356277    2563 generic.go:356] "Generic (PLEG): container finished" podID="ffbe4cc1-3536-4c5d-8032-c2ed53ecd183" containerID="d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c" exitCode=0
Sep 15 21:24:39.356560 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.356365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7" event={"ID":"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183","Type":"ContainerDied","Data":"d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c"}
Sep 15 21:24:39.356560 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.356394    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"
Sep 15 21:24:39.356560 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.356462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7" event={"ID":"ffbe4cc1-3536-4c5d-8032-c2ed53ecd183","Type":"ContainerDied","Data":"8813b0a0140b904dec204a6cf59e12e2f81557655cb0142215f6b594bbe831ae"}
Sep 15 21:24:39.356560 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.356483    2563 scope.go:117] "RemoveContainer" containerID="d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c"
Sep 15 21:24:39.358240 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.358215    2563 generic.go:356] "Generic (PLEG): container finished" podID="05f7e17c-98f6-4d47-8af5-4e457fde209e" containerID="c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e" exitCode=0
Sep 15 21:24:39.358536 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.358249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc" event={"ID":"05f7e17c-98f6-4d47-8af5-4e457fde209e","Type":"ContainerDied","Data":"c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e"}
Sep 15 21:24:39.358536 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.358265    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"
Sep 15 21:24:39.358536 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.358320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc" event={"ID":"05f7e17c-98f6-4d47-8af5-4e457fde209e","Type":"ContainerDied","Data":"e9753cc8b0bf761752c6ba04dee2f22a43f1638d62b9cd64d11f4ffc62f477a1"}
Sep 15 21:24:39.373597 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.373568    2563 scope.go:117] "RemoveContainer" containerID="d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c"
Sep 15 21:24:39.374137 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:24:39.374102    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c\": container with ID starting with d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c not found: ID does not exist" containerID="d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c"
Sep 15 21:24:39.374230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.374154    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c"} err="failed to get container status \"d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c\": rpc error: code = NotFound desc = could not find container \"d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c\": container with ID starting with d436f52ff8cf6ee5bf559b46935728253cbfafe8a7247476f80b0270e2d3663c not found: ID does not exist"
Sep 15 21:24:39.374230 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.374185    2563 scope.go:117] "RemoveContainer" containerID="c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e"
Sep 15 21:24:39.386584 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.386561    2563 scope.go:117] "RemoveContainer" containerID="c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e"
Sep 15 21:24:39.387180 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:24:39.387145    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e\": container with ID starting with c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e not found: ID does not exist" containerID="c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e"
Sep 15 21:24:39.387286 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.387182    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e"} err="failed to get container status \"c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e\": rpc error: code = NotFound desc = could not find container \"c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e\": container with ID starting with c0580ef7b7a45a3522ab9cf5cff6cab0a3d07f46120093ed6a32e29bb45a355e not found: ID does not exist"
Sep 15 21:24:39.398139 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.398104    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"]
Sep 15 21:24:39.414851 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.414811    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44fbc30a-d6cffb77-rzbh7"]
Sep 15 21:24:39.435694 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.435657    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"]
Sep 15 21:24:39.437883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.437860    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fbc30a-7d49cbbff4-vh2xc"]
Sep 15 21:24:39.904736 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.904707    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f7e17c-98f6-4d47-8af5-4e457fde209e" path="/var/lib/kubelet/pods/05f7e17c-98f6-4d47-8af5-4e457fde209e/volumes"
Sep 15 21:24:39.905109 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.905054    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2781e8-1e0b-4ca1-99a7-19db81bd195c" path="/var/lib/kubelet/pods/6f2781e8-1e0b-4ca1-99a7-19db81bd195c/volumes"
Sep 15 21:24:39.905365 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:39.905352    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffbe4cc1-3536-4c5d-8032-c2ed53ecd183" path="/var/lib/kubelet/pods/ffbe4cc1-3536-4c5d-8032-c2ed53ecd183/volumes"
Sep 15 21:24:41.341812 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:41.341781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:24:46.338394 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:46.338362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:24:49.389584 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.389547    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"]
Sep 15 21:24:49.390290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.390263    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbe4cc1-3536-4c5d-8032-c2ed53ecd183" containerName="payload-processing"
Sep 15 21:24:49.390335 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.390294    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbe4cc1-3536-4c5d-8032-c2ed53ecd183" containerName="payload-processing"
Sep 15 21:24:49.390335 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.390325    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f7e17c-98f6-4d47-8af5-4e457fde209e" containerName="payload-pre-processing"
Sep 15 21:24:49.390335 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.390333    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f7e17c-98f6-4d47-8af5-4e457fde209e" containerName="payload-pre-processing"
Sep 15 21:24:49.390474 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.390460    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f7e17c-98f6-4d47-8af5-4e457fde209e" containerName="payload-pre-processing"
Sep 15 21:24:49.390517 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.390478    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbe4cc1-3536-4c5d-8032-c2ed53ecd183" containerName="payload-processing"
Sep 15 21:24:49.395910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.395887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.398140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.398068    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b2f6c94e-serving-cert\""
Sep 15 21:24:49.409310 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.409279    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"]
Sep 15 21:24:49.476549 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.476515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqzjg\" (UniqueName: \"kubernetes.io/projected/13f7df38-cd2b-451a-9784-98022e3bc8e6-kube-api-access-cqzjg\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.476768 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.476632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.476768 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.476686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-metrics-certs\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.476768 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.476704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f7df38-cd2b-451a-9784-98022e3bc8e6-openshift-service-ca\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.577985 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.577941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.578207 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.578030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-metrics-certs\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.578207 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.578059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f7df38-cd2b-451a-9784-98022e3bc8e6-openshift-service-ca\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.578207 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:24:49.578097    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-serving-cert: secret "maas-api-e2e-migrate-b2f6c94e-serving-cert" not found
Sep 15 21:24:49.578207 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:24:49.578183    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls podName:13f7df38-cd2b-451a-9784-98022e3bc8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:24:50.07815931 +0000 UTC m=+1008.678071415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls") pod "maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv" (UID: "13f7df38-cd2b-451a-9784-98022e3bc8e6") : secret "maas-api-e2e-migrate-b2f6c94e-serving-cert" not found
Sep 15 21:24:49.578207 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.578103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqzjg\" (UniqueName: \"kubernetes.io/projected/13f7df38-cd2b-451a-9784-98022e3bc8e6-kube-api-access-cqzjg\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.578802 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.578783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f7df38-cd2b-451a-9784-98022e3bc8e6-openshift-service-ca\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.580711 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.580694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-metrics-certs\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:49.590376 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:49.590342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqzjg\" (UniqueName: \"kubernetes.io/projected/13f7df38-cd2b-451a-9784-98022e3bc8e6-kube-api-access-cqzjg\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:50.085060 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:50.085020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:50.087885 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:50.087857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls\") pod \"maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:50.311177 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:50.311137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:50.596482 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:50.596428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"]
Sep 15 21:24:50.599368 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:24:50.599329    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f7df38_cd2b_451a_9784_98022e3bc8e6.slice/crio-d4bf5fb689b3b499b2672c0a4140191edab4eeb22e75c47e9ef3d08ebf07a022 WatchSource:0}: Error finding container d4bf5fb689b3b499b2672c0a4140191edab4eeb22e75c47e9ef3d08ebf07a022: Status 404 returned error can't find the container with id d4bf5fb689b3b499b2672c0a4140191edab4eeb22e75c47e9ef3d08ebf07a022
Sep 15 21:24:51.447620 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:51.447583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv" event={"ID":"13f7df38-cd2b-451a-9784-98022e3bc8e6","Type":"ContainerStarted","Data":"92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c"}
Sep 15 21:24:51.447852 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:51.447672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv" event={"ID":"13f7df38-cd2b-451a-9784-98022e3bc8e6","Type":"ContainerStarted","Data":"d4bf5fb689b3b499b2672c0a4140191edab4eeb22e75c47e9ef3d08ebf07a022"}
Sep 15 21:24:51.447852 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:51.447724    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:24:51.473065 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:51.473007    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv" podStartSLOduration=2.150384684 podStartE2EDuration="2.472988651s" totalImagesPullingTime="322.603967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:51.471189222 +0000 UTC m=+1010.071101333" watchObservedRunningTime="2026-09-15 21:24:51.472988651 +0000 UTC m=+1010.072900760"
Sep 15 21:24:57.459379 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:24:57.459348    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:25:26.871150 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:26.870978    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"]
Sep 15 21:25:26.871729 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:26.871209    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv" podUID="13f7df38-cd2b-451a-9784-98022e3bc8e6" containerName="maas-api" containerID="cri-o://92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c" gracePeriod=30
Sep 15 21:25:27.137491 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.137417    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:25:27.260363 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.260327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqzjg\" (UniqueName: \"kubernetes.io/projected/13f7df38-cd2b-451a-9784-98022e3bc8e6-kube-api-access-cqzjg\") pod \"13f7df38-cd2b-451a-9784-98022e3bc8e6\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") "
Sep 15 21:25:27.260569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.260376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-metrics-certs\") pod \"13f7df38-cd2b-451a-9784-98022e3bc8e6\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") "
Sep 15 21:25:27.260569 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.260467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f7df38-cd2b-451a-9784-98022e3bc8e6-openshift-service-ca\") pod \"13f7df38-cd2b-451a-9784-98022e3bc8e6\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") "
Sep 15 21:25:27.260712 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.260571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls\") pod \"13f7df38-cd2b-451a-9784-98022e3bc8e6\" (UID: \"13f7df38-cd2b-451a-9784-98022e3bc8e6\") "
Sep 15 21:25:27.261057 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.261012    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f7df38-cd2b-451a-9784-98022e3bc8e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13f7df38-cd2b-451a-9784-98022e3bc8e6" (UID: "13f7df38-cd2b-451a-9784-98022e3bc8e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:27.263863 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.263823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "13f7df38-cd2b-451a-9784-98022e3bc8e6" (UID: "13f7df38-cd2b-451a-9784-98022e3bc8e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:25:27.263863 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.263842    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f7df38-cd2b-451a-9784-98022e3bc8e6-kube-api-access-cqzjg" (OuterVolumeSpecName: "kube-api-access-cqzjg") pod "13f7df38-cd2b-451a-9784-98022e3bc8e6" (UID: "13f7df38-cd2b-451a-9784-98022e3bc8e6"). InnerVolumeSpecName "kube-api-access-cqzjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:27.264043 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.263846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13f7df38-cd2b-451a-9784-98022e3bc8e6" (UID: "13f7df38-cd2b-451a-9784-98022e3bc8e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:25:27.362185 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.362157    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:25:27.362185 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.362182    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqzjg\" (UniqueName: \"kubernetes.io/projected/13f7df38-cd2b-451a-9784-98022e3bc8e6-kube-api-access-cqzjg\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:25:27.362185 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.362194    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f7df38-cd2b-451a-9784-98022e3bc8e6-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:25:27.362426 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.362202    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f7df38-cd2b-451a-9784-98022e3bc8e6-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:25:27.661844 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.661540    2563 generic.go:356] "Generic (PLEG): container finished" podID="13f7df38-cd2b-451a-9784-98022e3bc8e6" containerID="92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c" exitCode=0
Sep 15 21:25:27.661844 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.661684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv" event={"ID":"13f7df38-cd2b-451a-9784-98022e3bc8e6","Type":"ContainerDied","Data":"92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c"}
Sep 15 21:25:27.661844 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.661771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv" event={"ID":"13f7df38-cd2b-451a-9784-98022e3bc8e6","Type":"ContainerDied","Data":"d4bf5fb689b3b499b2672c0a4140191edab4eeb22e75c47e9ef3d08ebf07a022"}
Sep 15 21:25:27.661844 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.661804    2563 scope.go:117] "RemoveContainer" containerID="92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c"
Sep 15 21:25:27.662158 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.661912    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"
Sep 15 21:25:27.672580 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.672561    2563 scope.go:117] "RemoveContainer" containerID="92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c"
Sep 15 21:25:27.672880 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:25:27.672857    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c\": container with ID starting with 92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c not found: ID does not exist" containerID="92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c"
Sep 15 21:25:27.672939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.672892    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c"} err="failed to get container status \"92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c\": rpc error: code = NotFound desc = could not find container \"92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c\": container with ID starting with 92bebfa631729c3d53a6f89d024659ab45d4d1f6f49e7d439899ac6d1662691c not found: ID does not exist"
Sep 15 21:25:27.687266 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.687228    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"]
Sep 15 21:25:27.692024 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.692000    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2f6c94e-7c85bf558b-kldkv"]
Sep 15 21:25:27.903142 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:27.903101    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f7df38-cd2b-451a-9784-98022e3bc8e6" path="/var/lib/kubelet/pods/13f7df38-cd2b-451a-9784-98022e3bc8e6/volumes"
Sep 15 21:25:32.875320 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:32.875291    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"]
Sep 15 21:25:32.876280 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:32.875940    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f7df38-cd2b-451a-9784-98022e3bc8e6" containerName="maas-api"
Sep 15 21:25:32.876280 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:32.875964    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f7df38-cd2b-451a-9784-98022e3bc8e6" containerName="maas-api"
Sep 15 21:25:32.876280 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:32.876080    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f7df38-cd2b-451a-9784-98022e3bc8e6" containerName="maas-api"
Sep 15 21:25:32.879647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:32.879627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:32.881845 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:32.881830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d219b8ad-serving-cert\""
Sep 15 21:25:32.892962 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:32.892899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"]
Sep 15 21:25:33.019101 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.019059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmpk\" (UniqueName: \"kubernetes.io/projected/f3062bdf-418d-491e-960c-88a709583462-kube-api-access-jmmpk\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.019274 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.019178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-metrics-certs\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.019325 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.019264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3062bdf-418d-491e-960c-88a709583462-openshift-service-ca\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.019399 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.019381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.119991 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.119955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.120184 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.120026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmpk\" (UniqueName: \"kubernetes.io/projected/f3062bdf-418d-491e-960c-88a709583462-kube-api-access-jmmpk\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.120184 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:25:33.120088    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-serving-cert: secret "maas-api-e2e-ait-d219b8ad-serving-cert" not found
Sep 15 21:25:33.120184 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.120100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-metrics-certs\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.120184 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:25:33.120141    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls podName:f3062bdf-418d-491e-960c-88a709583462 nodeName:}" failed. No retries permitted until 2026-09-15 21:25:33.620125207 +0000 UTC m=+1052.220037300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls") pod "maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2" (UID: "f3062bdf-418d-491e-960c-88a709583462") : secret "maas-api-e2e-ait-d219b8ad-serving-cert" not found
Sep 15 21:25:33.120184 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.120157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3062bdf-418d-491e-960c-88a709583462-openshift-service-ca\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.120811 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.120793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3062bdf-418d-491e-960c-88a709583462-openshift-service-ca\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.122925 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.122896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-metrics-certs\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.129287 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.129239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmpk\" (UniqueName: \"kubernetes.io/projected/f3062bdf-418d-491e-960c-88a709583462-kube-api-access-jmmpk\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.626675 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.626641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.629225 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.629193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls\") pod \"maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:33.790952 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:33.790915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:34.068808 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:34.068785    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"]
Sep 15 21:25:34.070743 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:25:34.070713    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3062bdf_418d_491e_960c_88a709583462.slice/crio-49a6f05a9c493309ae1f68aea4e3a5f0cd3941d74528f757adec6e9e09c03ea7 WatchSource:0}: Error finding container 49a6f05a9c493309ae1f68aea4e3a5f0cd3941d74528f757adec6e9e09c03ea7: Status 404 returned error can't find the container with id 49a6f05a9c493309ae1f68aea4e3a5f0cd3941d74528f757adec6e9e09c03ea7
Sep 15 21:25:34.707040 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:34.707013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2" event={"ID":"f3062bdf-418d-491e-960c-88a709583462","Type":"ContainerStarted","Data":"e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74"}
Sep 15 21:25:34.707176 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:34.707063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2" event={"ID":"f3062bdf-418d-491e-960c-88a709583462","Type":"ContainerStarted","Data":"49a6f05a9c493309ae1f68aea4e3a5f0cd3941d74528f757adec6e9e09c03ea7"}
Sep 15 21:25:34.707176 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:34.707113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:25:34.747720 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:34.747654    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2" podStartSLOduration=2.194045694 podStartE2EDuration="2.747631713s" totalImagesPullingTime="553.586019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:34.74706255 +0000 UTC m=+1053.346974661" watchObservedRunningTime="2026-09-15 21:25:34.747631713 +0000 UTC m=+1053.347543825"
Sep 15 21:25:40.718123 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:25:40.718090    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:26:02.307752 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.307653    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"]
Sep 15 21:26:02.308256 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.307889    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2" podUID="f3062bdf-418d-491e-960c-88a709583462" containerName="maas-api" containerID="cri-o://e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74" gracePeriod=30
Sep 15 21:26:02.574313 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.574239    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:26:02.710594 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.710035    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-metrics-certs\") pod \"f3062bdf-418d-491e-960c-88a709583462\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") "
Sep 15 21:26:02.710594 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.710095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmmpk\" (UniqueName: \"kubernetes.io/projected/f3062bdf-418d-491e-960c-88a709583462-kube-api-access-jmmpk\") pod \"f3062bdf-418d-491e-960c-88a709583462\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") "
Sep 15 21:26:02.710594 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.710124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls\") pod \"f3062bdf-418d-491e-960c-88a709583462\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") "
Sep 15 21:26:02.710594 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.710158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3062bdf-418d-491e-960c-88a709583462-openshift-service-ca\") pod \"f3062bdf-418d-491e-960c-88a709583462\" (UID: \"f3062bdf-418d-491e-960c-88a709583462\") "
Sep 15 21:26:02.711983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.711946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3062bdf-418d-491e-960c-88a709583462-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3062bdf-418d-491e-960c-88a709583462" (UID: "f3062bdf-418d-491e-960c-88a709583462"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:02.714343 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.714308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3062bdf-418d-491e-960c-88a709583462" (UID: "f3062bdf-418d-491e-960c-88a709583462"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:02.714516 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.714399    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3062bdf-418d-491e-960c-88a709583462-kube-api-access-jmmpk" (OuterVolumeSpecName: "kube-api-access-jmmpk") pod "f3062bdf-418d-491e-960c-88a709583462" (UID: "f3062bdf-418d-491e-960c-88a709583462"). InnerVolumeSpecName "kube-api-access-jmmpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:02.715783 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.715758    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f3062bdf-418d-491e-960c-88a709583462" (UID: "f3062bdf-418d-491e-960c-88a709583462"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:02.812433 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.812400    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:02.812433 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.812428    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmpk\" (UniqueName: \"kubernetes.io/projected/f3062bdf-418d-491e-960c-88a709583462-kube-api-access-jmmpk\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:02.812433 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.812440    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3062bdf-418d-491e-960c-88a709583462-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:02.812730 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.812465    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3062bdf-418d-491e-960c-88a709583462-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:02.868097 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.868003    2563 generic.go:356] "Generic (PLEG): container finished" podID="f3062bdf-418d-491e-960c-88a709583462" containerID="e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74" exitCode=0
Sep 15 21:26:02.868097 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.868049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"
Sep 15 21:26:02.868282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.868099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2" event={"ID":"f3062bdf-418d-491e-960c-88a709583462","Type":"ContainerDied","Data":"e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74"}
Sep 15 21:26:02.868282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.868151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2" event={"ID":"f3062bdf-418d-491e-960c-88a709583462","Type":"ContainerDied","Data":"49a6f05a9c493309ae1f68aea4e3a5f0cd3941d74528f757adec6e9e09c03ea7"}
Sep 15 21:26:02.868282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.868176    2563 scope.go:117] "RemoveContainer" containerID="e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74"
Sep 15 21:26:02.877932 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.877915    2563 scope.go:117] "RemoveContainer" containerID="e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74"
Sep 15 21:26:02.878232 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:26:02.878211    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74\": container with ID starting with e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74 not found: ID does not exist" containerID="e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74"
Sep 15 21:26:02.878281 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.878241    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74"} err="failed to get container status \"e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74\": rpc error: code = NotFound desc = could not find container \"e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74\": container with ID starting with e8dca1c6b5baa48e5d0339ffc0454890ca676978d7ce5a6d0a25699ac45d0a74 not found: ID does not exist"
Sep 15 21:26:02.890676 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.890651    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"]
Sep 15 21:26:02.897288 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:02.897258    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d219b8ad-cf9d8c44c-p6jv2"]
Sep 15 21:26:03.903702 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:03.903673    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3062bdf-418d-491e-960c-88a709583462" path="/var/lib/kubelet/pods/f3062bdf-418d-491e-960c-88a709583462/volumes"
Sep 15 21:26:08.948298 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:08.948261    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"]
Sep 15 21:26:08.949017 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:08.948996    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3062bdf-418d-491e-960c-88a709583462" containerName="maas-api"
Sep 15 21:26:08.949017 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:08.949018    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3062bdf-418d-491e-960c-88a709583462" containerName="maas-api"
Sep 15 21:26:08.949180 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:08.949169    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3062bdf-418d-491e-960c-88a709583462" containerName="maas-api"
Sep 15 21:26:08.953822 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:08.953794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:08.956119 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:08.956100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-639cece9-serving-cert\""
Sep 15 21:26:08.960620 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:08.960598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"]
Sep 15 21:26:09.072248 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.072182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-openshift-service-ca\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.072248 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.072234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkw48\" (UniqueName: \"kubernetes.io/projected/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-kube-api-access-dkw48\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.072670 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.072364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-metrics-certs\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.072670 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.072472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-maas-api-tls\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.173256 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.173220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-metrics-certs\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.173432 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.173272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-maas-api-tls\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.173432 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.173298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-openshift-service-ca\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.173432 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.173317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkw48\" (UniqueName: \"kubernetes.io/projected/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-kube-api-access-dkw48\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.173997 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.173972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-openshift-service-ca\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.175712 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.175687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-metrics-certs\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.175853 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.175833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-maas-api-tls\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.182050 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.182027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkw48\" (UniqueName: \"kubernetes.io/projected/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-kube-api-access-dkw48\") pod \"maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.267871 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.267839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:09.546423 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.546399    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"]
Sep 15 21:26:09.548947 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:26:09.548911    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f3f51f_4091_4c27_9e1b_3e8dd195f1f8.slice/crio-ad6d695cd70de4c5fd5ed1853202b97c7efcfcaf44b4cf9337daa49c271aca38 WatchSource:0}: Error finding container ad6d695cd70de4c5fd5ed1853202b97c7efcfcaf44b4cf9337daa49c271aca38: Status 404 returned error can't find the container with id ad6d695cd70de4c5fd5ed1853202b97c7efcfcaf44b4cf9337daa49c271aca38
Sep 15 21:26:09.914558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:09.914476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx" event={"ID":"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8","Type":"ContainerStarted","Data":"ad6d695cd70de4c5fd5ed1853202b97c7efcfcaf44b4cf9337daa49c271aca38"}
Sep 15 21:26:10.922127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:10.922088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx" event={"ID":"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8","Type":"ContainerStarted","Data":"efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19"}
Sep 15 21:26:10.922517 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:10.922174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:10.941555 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:10.941505    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx" podStartSLOduration=2.502327092 podStartE2EDuration="2.941489313s" totalImagesPullingTime="439.162221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:10.938472489 +0000 UTC m=+1089.538384600" watchObservedRunningTime="2026-09-15 21:26:10.941489313 +0000 UTC m=+1089.541401422"
Sep 15 21:26:16.932785 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:16.932752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:39.040157 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.040123    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"]
Sep 15 21:26:39.041144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.041108    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx" podUID="39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" containerName="maas-api" containerID="cri-o://efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19" gracePeriod=30
Sep 15 21:26:39.309811 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.309728    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:39.354098 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.354069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-openshift-service-ca\") pod \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") "
Sep 15 21:26:39.354280 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.354218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkw48\" (UniqueName: \"kubernetes.io/projected/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-kube-api-access-dkw48\") pod \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") "
Sep 15 21:26:39.354280 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.354252    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-metrics-certs\") pod \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") "
Sep 15 21:26:39.354412 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.354307    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-maas-api-tls\") pod \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\" (UID: \"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8\") "
Sep 15 21:26:39.354524 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.354492    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" (UID: "39f3f51f-4091-4c27-9e1b-3e8dd195f1f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:39.354682 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.354661    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:39.356723 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.356688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" (UID: "39f3f51f-4091-4c27-9e1b-3e8dd195f1f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:39.357043 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.357011    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" (UID: "39f3f51f-4091-4c27-9e1b-3e8dd195f1f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:39.357125 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.357042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-kube-api-access-dkw48" (OuterVolumeSpecName: "kube-api-access-dkw48") pod "39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" (UID: "39f3f51f-4091-4c27-9e1b-3e8dd195f1f8"). InnerVolumeSpecName "kube-api-access-dkw48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:39.456249 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.456209    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkw48\" (UniqueName: \"kubernetes.io/projected/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-kube-api-access-dkw48\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:39.456249 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.456246    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:39.456502 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:39.456260    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:40.092777 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.092742    2563 generic.go:356] "Generic (PLEG): container finished" podID="39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" containerID="efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19" exitCode=0
Sep 15 21:26:40.093394 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.092820    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"
Sep 15 21:26:40.093394 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.092843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx" event={"ID":"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8","Type":"ContainerDied","Data":"efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19"}
Sep 15 21:26:40.093394 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.092903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx" event={"ID":"39f3f51f-4091-4c27-9e1b-3e8dd195f1f8","Type":"ContainerDied","Data":"ad6d695cd70de4c5fd5ed1853202b97c7efcfcaf44b4cf9337daa49c271aca38"}
Sep 15 21:26:40.093394 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.092928    2563 scope.go:117] "RemoveContainer" containerID="efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19"
Sep 15 21:26:40.105192 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.105160    2563 scope.go:117] "RemoveContainer" containerID="efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19"
Sep 15 21:26:40.105622 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:26:40.105601    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19\": container with ID starting with efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19 not found: ID does not exist" containerID="efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19"
Sep 15 21:26:40.105739 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.105631    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19"} err="failed to get container status \"efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19\": rpc error: code = NotFound desc = could not find container \"efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19\": container with ID starting with efccbb646e06b6308dcdd2ce74df221d931a630dbef144247e77e032b4ea6e19 not found: ID does not exist"
Sep 15 21:26:40.176707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.176583    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"]
Sep 15 21:26:40.180987 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:40.180897    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-639cece9-7dc88d9778-ldmvx"]
Sep 15 21:26:41.905565 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:41.905526    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" path="/var/lib/kubelet/pods/39f3f51f-4091-4c27-9e1b-3e8dd195f1f8/volumes"
Sep 15 21:26:52.746016 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:52.745983    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"]
Sep 15 21:26:52.746534 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:52.746264    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5" podUID="2196d395-2660-4ec0-91d9-05331f52e144" containerName="payload-pre-processing" containerID="cri-o://8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315" gracePeriod=30
Sep 15 21:26:53.024933 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.024907    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:26:53.166978 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.166942    2563 generic.go:356] "Generic (PLEG): container finished" podID="2196d395-2660-4ec0-91d9-05331f52e144" containerID="8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315" exitCode=0
Sep 15 21:26:53.167151 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.166988    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"
Sep 15 21:26:53.167151 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.166996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5" event={"ID":"2196d395-2660-4ec0-91d9-05331f52e144","Type":"ContainerDied","Data":"8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315"}
Sep 15 21:26:53.167151 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.167041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5" event={"ID":"2196d395-2660-4ec0-91d9-05331f52e144","Type":"ContainerDied","Data":"7f608602c004096b9ce64cfe22d2bde8db7fe08884d0292fcbce0950f88174be"}
Sep 15 21:26:53.167151 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.167063    2563 scope.go:117] "RemoveContainer" containerID="8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315"
Sep 15 21:26:53.176087 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.176070    2563 scope.go:117] "RemoveContainer" containerID="8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315"
Sep 15 21:26:53.176349 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:26:53.176334    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315\": container with ID starting with 8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315 not found: ID does not exist" containerID="8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315"
Sep 15 21:26:53.176404 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.176358    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315"} err="failed to get container status \"8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315\": rpc error: code = NotFound desc = could not find container \"8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315\": container with ID starting with 8be5f16888584a7610d998eedf1086a3a7a596a509ebe63a71c2c0c0cb961315 not found: ID does not exist"
Sep 15 21:26:53.198739 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.198719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqgcv\" (UniqueName: \"kubernetes.io/projected/2196d395-2660-4ec0-91d9-05331f52e144-kube-api-access-bqgcv\") pod \"2196d395-2660-4ec0-91d9-05331f52e144\" (UID: \"2196d395-2660-4ec0-91d9-05331f52e144\") "
Sep 15 21:26:53.198880 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.198797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2196d395-2660-4ec0-91d9-05331f52e144-plugins-config-volume\") pod \"2196d395-2660-4ec0-91d9-05331f52e144\" (UID: \"2196d395-2660-4ec0-91d9-05331f52e144\") "
Sep 15 21:26:53.199186 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.199160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2196d395-2660-4ec0-91d9-05331f52e144-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2196d395-2660-4ec0-91d9-05331f52e144" (UID: "2196d395-2660-4ec0-91d9-05331f52e144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:53.200945 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.200923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2196d395-2660-4ec0-91d9-05331f52e144-kube-api-access-bqgcv" (OuterVolumeSpecName: "kube-api-access-bqgcv") pod "2196d395-2660-4ec0-91d9-05331f52e144" (UID: "2196d395-2660-4ec0-91d9-05331f52e144"). InnerVolumeSpecName "kube-api-access-bqgcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:53.300069 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.299988    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2196d395-2660-4ec0-91d9-05331f52e144-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:53.300069 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.300024    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgcv\" (UniqueName: \"kubernetes.io/projected/2196d395-2660-4ec0-91d9-05331f52e144-kube-api-access-bqgcv\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:26:53.493917 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.493886    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"]
Sep 15 21:26:53.496368 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.496337    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-faa4c5-85d4f9bf74-ch9n5"]
Sep 15 21:26:53.902584 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:26:53.902542    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2196d395-2660-4ec0-91d9-05331f52e144" path="/var/lib/kubelet/pods/2196d395-2660-4ec0-91d9-05331f52e144/volumes"
Sep 15 21:27:42.347623 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.347593    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"]
Sep 15 21:27:42.348249 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.348223    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" containerName="maas-api"
Sep 15 21:27:42.348249 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.348250    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" containerName="maas-api"
Sep 15 21:27:42.348522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.348294    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2196d395-2660-4ec0-91d9-05331f52e144" containerName="payload-pre-processing"
Sep 15 21:27:42.348522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.348304    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2196d395-2660-4ec0-91d9-05331f52e144" containerName="payload-pre-processing"
Sep 15 21:27:42.348522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.348407    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f3f51f-4091-4c27-9e1b-3e8dd195f1f8" containerName="maas-api"
Sep 15 21:27:42.348522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.348424    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2196d395-2660-4ec0-91d9-05331f52e144" containerName="payload-pre-processing"
Sep 15 21:27:42.352322 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.352304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.354510 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.354482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-928184-serving-cert\""
Sep 15 21:27:42.362501 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.362205    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"]
Sep 15 21:27:42.390477 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.390419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.390678 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.390503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-metrics-certs\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.390678 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.390538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87qh5\" (UniqueName: \"kubernetes.io/projected/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-kube-api-access-87qh5\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.390810 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.390682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.492026 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.491983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.492246 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.492037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.492246 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.492070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-metrics-certs\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.492246 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.492100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87qh5\" (UniqueName: \"kubernetes.io/projected/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-kube-api-access-87qh5\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.492246 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:27:42.492209    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-928184-serving-cert: secret "maas-api-e2e-explicit-928184-serving-cert" not found
Sep 15 21:27:42.492499 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:27:42.492313    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls podName:c9c4b554-4583-46c7-a2d4-8e39a26dc5ee nodeName:}" failed. No retries permitted until 2026-09-15 21:27:42.992291532 +0000 UTC m=+1181.592203643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls") pod "maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9" (UID: "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee") : secret "maas-api-e2e-explicit-928184-serving-cert" not found
Sep 15 21:27:42.493155 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.493132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.494892 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.494866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-metrics-certs\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.504533 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.504503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87qh5\" (UniqueName: \"kubernetes.io/projected/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-kube-api-access-87qh5\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:42.998933 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:42.998896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:43.001776 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:43.001637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls\") pod \"maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:43.265366 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:43.265281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:43.554234 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:43.554210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"]
Sep 15 21:27:43.555903 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:27:43.555876    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c4b554_4583_46c7_a2d4_8e39a26dc5ee.slice/crio-678e27f2db815748af1904c4237c83fb732b0615e37b04a5f058e70e31d723e3 WatchSource:0}: Error finding container 678e27f2db815748af1904c4237c83fb732b0615e37b04a5f058e70e31d723e3: Status 404 returned error can't find the container with id 678e27f2db815748af1904c4237c83fb732b0615e37b04a5f058e70e31d723e3
Sep 15 21:27:44.468906 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:44.468871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9" event={"ID":"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee","Type":"ContainerStarted","Data":"0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e"}
Sep 15 21:27:44.469092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:44.468920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9" event={"ID":"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee","Type":"ContainerStarted","Data":"678e27f2db815748af1904c4237c83fb732b0615e37b04a5f058e70e31d723e3"}
Sep 15 21:27:44.469092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:44.468983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:27:44.486744 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:44.486705    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9" podStartSLOduration=1.997961082 podStartE2EDuration="2.486691433s" totalImagesPullingTime="488.730351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:44.484781415 +0000 UTC m=+1183.084693525" watchObservedRunningTime="2026-09-15 21:27:44.486691433 +0000 UTC m=+1183.086603570"
Sep 15 21:27:50.480425 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:27:50.480395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:28:27.782342 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.782300    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"]
Sep 15 21:28:27.786563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.786541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.788628 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.788608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-11b5978b-serving-cert\""
Sep 15 21:28:27.797667 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.796146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"]
Sep 15 21:28:27.832659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.832629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1169e0-1e82-4b3a-80b4-55d06544a965-openshift-service-ca\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.832659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.832658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-metrics-certs\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.832864 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.832784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.832864 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.832829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdq5r\" (UniqueName: \"kubernetes.io/projected/fb1169e0-1e82-4b3a-80b4-55d06544a965-kube-api-access-sdq5r\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.933579 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.933542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.933753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.933597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdq5r\" (UniqueName: \"kubernetes.io/projected/fb1169e0-1e82-4b3a-80b4-55d06544a965-kube-api-access-sdq5r\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.933753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.933654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1169e0-1e82-4b3a-80b4-55d06544a965-openshift-service-ca\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.933753 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.933672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-metrics-certs\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.933753 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:28:27.933694    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-serving-cert: secret "maas-api-e2e-mt-11b5978b-serving-cert" not found
Sep 15 21:28:27.933899 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:28:27.933762    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls podName:fb1169e0-1e82-4b3a-80b4-55d06544a965 nodeName:}" failed. No retries permitted until 2026-09-15 21:28:28.433741916 +0000 UTC m=+1227.033654018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls") pod "maas-api-e2e-mt-11b5978b-7664866c48-r2kvt" (UID: "fb1169e0-1e82-4b3a-80b4-55d06544a965") : secret "maas-api-e2e-mt-11b5978b-serving-cert" not found
Sep 15 21:28:27.934336 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.934315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1169e0-1e82-4b3a-80b4-55d06544a965-openshift-service-ca\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.936235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.936213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-metrics-certs\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:27.944568 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:27.944548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdq5r\" (UniqueName: \"kubernetes.io/projected/fb1169e0-1e82-4b3a-80b4-55d06544a965-kube-api-access-sdq5r\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:28.439748 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:28.439709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:28.442275 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:28.442239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls\") pod \"maas-api-e2e-mt-11b5978b-7664866c48-r2kvt\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:28.702163 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:28.702073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:28.975479 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:28.975439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"]
Sep 15 21:28:28.977430 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:28:28.977404    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1169e0_1e82_4b3a_80b4_55d06544a965.slice/crio-80c6b3a7b1814518a976504a718a3ef0ae9cc4d6e8fd05e98c4e53d099a0da32 WatchSource:0}: Error finding container 80c6b3a7b1814518a976504a718a3ef0ae9cc4d6e8fd05e98c4e53d099a0da32: Status 404 returned error can't find the container with id 80c6b3a7b1814518a976504a718a3ef0ae9cc4d6e8fd05e98c4e53d099a0da32
Sep 15 21:28:29.729862 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:29.729831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt" event={"ID":"fb1169e0-1e82-4b3a-80b4-55d06544a965","Type":"ContainerStarted","Data":"80c6b3a7b1814518a976504a718a3ef0ae9cc4d6e8fd05e98c4e53d099a0da32"}
Sep 15 21:28:30.736710 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:30.736665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt" event={"ID":"fb1169e0-1e82-4b3a-80b4-55d06544a965","Type":"ContainerStarted","Data":"69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e"}
Sep 15 21:28:30.737099 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:30.736741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:28:30.755770 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:30.755726    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt" podStartSLOduration=2.419765656 podStartE2EDuration="3.755711494s" totalImagesPullingTime="1.335945838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:30.754352553 +0000 UTC m=+1229.354264664" watchObservedRunningTime="2026-09-15 21:28:30.755711494 +0000 UTC m=+1229.355623628"
Sep 15 21:28:36.747912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:28:36.747880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:29:12.273271 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.273181    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"]
Sep 15 21:29:12.273899 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.273439    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt" podUID="fb1169e0-1e82-4b3a-80b4-55d06544a965" containerName="maas-api" containerID="cri-o://69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e" gracePeriod=30
Sep 15 21:29:12.535410 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.535345    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:29:12.677587 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.677557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-metrics-certs\") pod \"fb1169e0-1e82-4b3a-80b4-55d06544a965\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") "
Sep 15 21:29:12.677770 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.677625    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls\") pod \"fb1169e0-1e82-4b3a-80b4-55d06544a965\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") "
Sep 15 21:29:12.677770 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.677692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdq5r\" (UniqueName: \"kubernetes.io/projected/fb1169e0-1e82-4b3a-80b4-55d06544a965-kube-api-access-sdq5r\") pod \"fb1169e0-1e82-4b3a-80b4-55d06544a965\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") "
Sep 15 21:29:12.677770 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.677747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1169e0-1e82-4b3a-80b4-55d06544a965-openshift-service-ca\") pod \"fb1169e0-1e82-4b3a-80b4-55d06544a965\" (UID: \"fb1169e0-1e82-4b3a-80b4-55d06544a965\") "
Sep 15 21:29:12.678226 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.678189    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1169e0-1e82-4b3a-80b4-55d06544a965-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb1169e0-1e82-4b3a-80b4-55d06544a965" (UID: "fb1169e0-1e82-4b3a-80b4-55d06544a965"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:29:12.680036 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.680005    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1169e0-1e82-4b3a-80b4-55d06544a965-kube-api-access-sdq5r" (OuterVolumeSpecName: "kube-api-access-sdq5r") pod "fb1169e0-1e82-4b3a-80b4-55d06544a965" (UID: "fb1169e0-1e82-4b3a-80b4-55d06544a965"). InnerVolumeSpecName "kube-api-access-sdq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:12.680144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.680052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fb1169e0-1e82-4b3a-80b4-55d06544a965" (UID: "fb1169e0-1e82-4b3a-80b4-55d06544a965"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:29:12.680355 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.680334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb1169e0-1e82-4b3a-80b4-55d06544a965" (UID: "fb1169e0-1e82-4b3a-80b4-55d06544a965"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:29:12.779258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.779213    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdq5r\" (UniqueName: \"kubernetes.io/projected/fb1169e0-1e82-4b3a-80b4-55d06544a965-kube-api-access-sdq5r\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:29:12.779258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.779250    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1169e0-1e82-4b3a-80b4-55d06544a965-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:29:12.779258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.779265    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:29:12.779549 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.779277    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1169e0-1e82-4b3a-80b4-55d06544a965-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:29:12.983589 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.983552    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb1169e0-1e82-4b3a-80b4-55d06544a965" containerID="69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e" exitCode=0
Sep 15 21:29:12.983589 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.983588    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"
Sep 15 21:29:12.983813 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.983594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt" event={"ID":"fb1169e0-1e82-4b3a-80b4-55d06544a965","Type":"ContainerDied","Data":"69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e"}
Sep 15 21:29:12.983813 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.983643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt" event={"ID":"fb1169e0-1e82-4b3a-80b4-55d06544a965","Type":"ContainerDied","Data":"80c6b3a7b1814518a976504a718a3ef0ae9cc4d6e8fd05e98c4e53d099a0da32"}
Sep 15 21:29:12.983813 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.983666    2563 scope.go:117] "RemoveContainer" containerID="69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e"
Sep 15 21:29:12.994506 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.994487    2563 scope.go:117] "RemoveContainer" containerID="69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e"
Sep 15 21:29:12.994814 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:29:12.994783    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e\": container with ID starting with 69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e not found: ID does not exist" containerID="69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e"
Sep 15 21:29:12.994918 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:12.994822    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e"} err="failed to get container status \"69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e\": rpc error: code = NotFound desc = could not find container \"69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e\": container with ID starting with 69d98ce168332d82d897174be69a3d565a0938e3d204093bb421f9ca5c5aae1e not found: ID does not exist"
Sep 15 21:29:13.004807 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:13.004787    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"]
Sep 15 21:29:13.008938 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:13.008916    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11b5978b-7664866c48-r2kvt"]
Sep 15 21:29:13.901459 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:13.901422    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1169e0-1e82-4b3a-80b4-55d06544a965" path="/var/lib/kubelet/pods/fb1169e0-1e82-4b3a-80b4-55d06544a965/volumes"
Sep 15 21:29:27.819617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.819586    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"]
Sep 15 21:29:27.820236 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.820216    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1169e0-1e82-4b3a-80b4-55d06544a965" containerName="maas-api"
Sep 15 21:29:27.820333 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.820239    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1169e0-1e82-4b3a-80b4-55d06544a965" containerName="maas-api"
Sep 15 21:29:27.820387 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.820353    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1169e0-1e82-4b3a-80b4-55d06544a965" containerName="maas-api"
Sep 15 21:29:27.825038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.825018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:27.827164 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.827139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"openshift-service-ca.crt\""
Sep 15 21:29:27.827868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.827833    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"distinct-w0-a276f9-kserve-self-signed-certs\""
Sep 15 21:29:27.827868 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.827837    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"kube-root-ca.crt\""
Sep 15 21:29:27.828052 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.827934    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"default-dockercfg-ksdmf\""
Sep 15 21:29:27.838227 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.838204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"]
Sep 15 21:29:27.915062 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.915029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-home\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:27.915062 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.915057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-model-cache\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:27.915282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.915083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-dshm\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:27.915282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.915224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tls-certs\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:27.915282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.915249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb9dx\" (UniqueName: \"kubernetes.io/projected/4186d9f9-39af-418d-8c62-26d8a81bb6fa-kube-api-access-wb9dx\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:27.915470 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:27.915339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tmp-dir\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016231 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-home\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016231 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-model-cache\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016491 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-dshm\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016491 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tls-certs\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016491 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9dx\" (UniqueName: \"kubernetes.io/projected/4186d9f9-39af-418d-8c62-26d8a81bb6fa-kube-api-access-wb9dx\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016491 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tmp-dir\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-home\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016762 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-model-cache\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.016807 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.016768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tmp-dir\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.018745 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.018719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-dshm\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.019005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.018987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tls-certs\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.026939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.026913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9dx\" (UniqueName: \"kubernetes.io/projected/4186d9f9-39af-418d-8c62-26d8a81bb6fa-kube-api-access-wb9dx\") pod \"distinct-w0-a276f9-kserve-77bc4668cb-zcgfc\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.137928 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.137838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:28.417472 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:28.417433    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"]
Sep 15 21:29:28.419412 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:29:28.419374    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4186d9f9_39af_418d_8c62_26d8a81bb6fa.slice/crio-7620a5a1eb7e447957278ce72f06c3b17b338cdddc7a9adacbd33c7b8c08157a WatchSource:0}: Error finding container 7620a5a1eb7e447957278ce72f06c3b17b338cdddc7a9adacbd33c7b8c08157a: Status 404 returned error can't find the container with id 7620a5a1eb7e447957278ce72f06c3b17b338cdddc7a9adacbd33c7b8c08157a
Sep 15 21:29:29.083398 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:29.083304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc" event={"ID":"4186d9f9-39af-418d-8c62-26d8a81bb6fa","Type":"ContainerStarted","Data":"60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b"}
Sep 15 21:29:29.083398 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:29.083359    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc" event={"ID":"4186d9f9-39af-418d-8c62-26d8a81bb6fa","Type":"ContainerStarted","Data":"7620a5a1eb7e447957278ce72f06c3b17b338cdddc7a9adacbd33c7b8c08157a"}
Sep 15 21:29:29.103258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:29.103209    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc" podStartSLOduration=2.103193096 podStartE2EDuration="2.103193096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:29.101411103 +0000 UTC m=+1287.701323213" watchObservedRunningTime="2026-09-15 21:29:29.103193096 +0000 UTC m=+1287.703105206"
Sep 15 21:29:38.138134 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:38.138080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:38.138642 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:38.138152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:38.151407 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:38.151379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:39.151371 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:39.151344    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:29:43.191143 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.191105    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"]
Sep 15 21:29:43.195576 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.195552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.197794 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.197773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-a3aa55-serving-cert\""
Sep 15 21:29:43.204660 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.204637    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"]
Sep 15 21:29:43.258883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.258851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b182b-50a4-4af9-a218-49a775acde60-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.259066 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.258899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jzd\" (UniqueName: \"kubernetes.io/projected/8b4b182b-50a4-4af9-a218-49a775acde60-kube-api-access-76jzd\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.259066 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.258934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.259066 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.258985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.360397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.360361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b182b-50a4-4af9-a218-49a775acde60-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.360582 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.360414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jzd\" (UniqueName: \"kubernetes.io/projected/8b4b182b-50a4-4af9-a218-49a775acde60-kube-api-access-76jzd\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.360582 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.360462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.360582 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.360512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.361013 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.360986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b182b-50a4-4af9-a218-49a775acde60-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.363102 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.363079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.363317 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.363298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.369954 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.369934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jzd\" (UniqueName: \"kubernetes.io/projected/8b4b182b-50a4-4af9-a218-49a775acde60-kube-api-access-76jzd\") pod \"maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.510101 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.510072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:43.797349 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.797316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"]
Sep 15 21:29:43.798631 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:29:43.798598    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4b182b_50a4_4af9_a218_49a775acde60.slice/crio-af1c745ef0e27c2079fddf380390833dfea927c404ecec7530c7671d296fa579 WatchSource:0}: Error finding container af1c745ef0e27c2079fddf380390833dfea927c404ecec7530c7671d296fa579: Status 404 returned error can't find the container with id af1c745ef0e27c2079fddf380390833dfea927c404ecec7530c7671d296fa579
Sep 15 21:29:43.800020 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:43.800002    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:29:44.054539 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.054381    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"]
Sep 15 21:29:44.054739 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.054646    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9" podUID="c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" containerName="maas-api" containerID="cri-o://0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e" gracePeriod=30
Sep 15 21:29:44.170636 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.170604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz" event={"ID":"8b4b182b-50a4-4af9-a218-49a775acde60","Type":"ContainerStarted","Data":"af1c745ef0e27c2079fddf380390833dfea927c404ecec7530c7671d296fa579"}
Sep 15 21:29:44.326933 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.326913    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:29:44.372010 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.371887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls\") pod \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") "
Sep 15 21:29:44.372010 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.371997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-metrics-certs\") pod \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") "
Sep 15 21:29:44.372227 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.372107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-openshift-service-ca\") pod \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") "
Sep 15 21:29:44.372561 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.372527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" (UID: "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:29:44.372662 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.372140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87qh5\" (UniqueName: \"kubernetes.io/projected/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-kube-api-access-87qh5\") pod \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\" (UID: \"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee\") "
Sep 15 21:29:44.374669 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.374426    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:29:44.375035 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.374992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-kube-api-access-87qh5" (OuterVolumeSpecName: "kube-api-access-87qh5") pod "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" (UID: "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee"). InnerVolumeSpecName "kube-api-access-87qh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:44.375383 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.375350    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" (UID: "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:29:44.377259 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.377239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" (UID: "c9c4b554-4583-46c7-a2d4-8e39a26dc5ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:29:44.476321 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.476286    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87qh5\" (UniqueName: \"kubernetes.io/projected/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-kube-api-access-87qh5\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:29:44.476321 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.476323    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:29:44.476756 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:44.476337    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:29:45.177742 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.177712    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" containerID="0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e" exitCode=0
Sep 15 21:29:45.177927 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.177760    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"
Sep 15 21:29:45.177927 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.177785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9" event={"ID":"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee","Type":"ContainerDied","Data":"0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e"}
Sep 15 21:29:45.177927 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.177828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9" event={"ID":"c9c4b554-4583-46c7-a2d4-8e39a26dc5ee","Type":"ContainerDied","Data":"678e27f2db815748af1904c4237c83fb732b0615e37b04a5f058e70e31d723e3"}
Sep 15 21:29:45.177927 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.177849    2563 scope.go:117] "RemoveContainer" containerID="0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e"
Sep 15 21:29:45.179784 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.179763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz" event={"ID":"8b4b182b-50a4-4af9-a218-49a775acde60","Type":"ContainerStarted","Data":"52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e"}
Sep 15 21:29:45.179907 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.179826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:29:45.187780 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.187765    2563 scope.go:117] "RemoveContainer" containerID="0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e"
Sep 15 21:29:45.188048 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:29:45.188031    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e\": container with ID starting with 0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e not found: ID does not exist" containerID="0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e"
Sep 15 21:29:45.188098 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.188055    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e"} err="failed to get container status \"0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e\": rpc error: code = NotFound desc = could not find container \"0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e\": container with ID starting with 0486b7ebf92b4a3a1b59add97a89c5cb6c5432dfb0c484970126be4d4f4dd07e not found: ID does not exist"
Sep 15 21:29:45.200220 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.200164    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz" podStartSLOduration=1.701798303 podStartE2EDuration="2.200151253s" totalImagesPullingTime="498.35295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:45.198012293 +0000 UTC m=+1303.797924403" watchObservedRunningTime="2026-09-15 21:29:45.200151253 +0000 UTC m=+1303.800063364"
Sep 15 21:29:45.219003 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.218974    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"]
Sep 15 21:29:45.235655 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.235630    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-928184-64f9cdd9d6-hwgt9"]
Sep 15 21:29:45.901525 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:45.901438    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" path="/var/lib/kubelet/pods/c9c4b554-4583-46c7-a2d4-8e39a26dc5ee/volumes"
Sep 15 21:29:51.192089 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:29:51.192056    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:30:00.237862 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.237824    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"]
Sep 15 21:30:00.238501 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.238485    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" containerName="maas-api"
Sep 15 21:30:00.238501 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.238503    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" containerName="maas-api"
Sep 15 21:30:00.238601 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.238581    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c4b554-4583-46c7-a2d4-8e39a26dc5ee" containerName="maas-api"
Sep 15 21:30:00.241768 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.241754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"
Sep 15 21:30:00.247588 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.247563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7fhv\""
Sep 15 21:30:00.253091 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.253070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"
Sep 15 21:30:00.259631 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.259611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"]
Sep 15 21:30:00.538817 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:00.538628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"]
Sep 15 21:30:00.541496 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:30:00.541467    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3dcb0d_bf32_4edd_8656_6dbf6e715aef.slice/crio-1a02e8ebf595ac1f332a4267202614756bebf0940ec141d561a67e008eb30309 WatchSource:0}: Error finding container 1a02e8ebf595ac1f332a4267202614756bebf0940ec141d561a67e008eb30309: Status 404 returned error can't find the container with id 1a02e8ebf595ac1f332a4267202614756bebf0940ec141d561a67e008eb30309
Sep 15 21:30:01.302647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:01.302614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z" event={"ID":"0f3dcb0d-bf32-4edd-8656-6dbf6e715aef","Type":"ContainerStarted","Data":"39143ebf0905d8184f0308de8aef8a34d91b14f9a93e19b2be0b68dad1448b2b"}
Sep 15 21:30:01.302996 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:01.302665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z" event={"ID":"0f3dcb0d-bf32-4edd-8656-6dbf6e715aef","Type":"ContainerStarted","Data":"1a02e8ebf595ac1f332a4267202614756bebf0940ec141d561a67e008eb30309"}
Sep 15 21:30:01.318367 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:01.318319    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z" podStartSLOduration=1.318301621 podStartE2EDuration="1.318301621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:01.317581733 +0000 UTC m=+1319.917493843" watchObservedRunningTime="2026-09-15 21:30:01.318301621 +0000 UTC m=+1319.918213730"
Sep 15 21:30:02.309106 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:02.309019    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f3dcb0d-bf32-4edd-8656-6dbf6e715aef" containerID="39143ebf0905d8184f0308de8aef8a34d91b14f9a93e19b2be0b68dad1448b2b" exitCode=0
Sep 15 21:30:02.309106 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:02.309055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z" event={"ID":"0f3dcb0d-bf32-4edd-8656-6dbf6e715aef","Type":"ContainerDied","Data":"39143ebf0905d8184f0308de8aef8a34d91b14f9a93e19b2be0b68dad1448b2b"}
Sep 15 21:30:03.450038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:03.450010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"
Sep 15 21:30:04.321094 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:04.321062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z" event={"ID":"0f3dcb0d-bf32-4edd-8656-6dbf6e715aef","Type":"ContainerDied","Data":"1a02e8ebf595ac1f332a4267202614756bebf0940ec141d561a67e008eb30309"}
Sep 15 21:30:04.321262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:04.321136    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a02e8ebf595ac1f332a4267202614756bebf0940ec141d561a67e008eb30309"
Sep 15 21:30:04.321262 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:04.321076    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"
Sep 15 21:30:16.872811 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.872777    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"]
Sep 15 21:30:16.873485 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.873468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f3dcb0d-bf32-4edd-8656-6dbf6e715aef" containerName="cleanup"
Sep 15 21:30:16.873543 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.873489    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f3dcb0d-bf32-4edd-8656-6dbf6e715aef" containerName="cleanup"
Sep 15 21:30:16.873637 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.873627    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f3dcb0d-bf32-4edd-8656-6dbf6e715aef" containerName="cleanup"
Sep 15 21:30:16.877401 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.877377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:16.879407 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.879389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-9aeead-serving-cert\""
Sep 15 21:30:16.885761 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.885336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"]
Sep 15 21:30:16.986984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.986945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q54m\" (UniqueName: \"kubernetes.io/projected/e4424a83-cdaf-4cd0-a403-55621bc9938d-kube-api-access-5q54m\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:16.987181 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.987037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:16.987181 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.987064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4424a83-cdaf-4cd0-a403-55621bc9938d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:16.987181 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:16.987079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.088290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.088259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q54m\" (UniqueName: \"kubernetes.io/projected/e4424a83-cdaf-4cd0-a403-55621bc9938d-kube-api-access-5q54m\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.088470 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.088315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.088470 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.088339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4424a83-cdaf-4cd0-a403-55621bc9938d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.088470 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.088357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.088577 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:30:17.088495    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-serving-cert: secret "maas-api-e2e-shared-b-w2-9aeead-serving-cert" not found
Sep 15 21:30:17.088618 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:30:17.088581    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls podName:e4424a83-cdaf-4cd0-a403-55621bc9938d nodeName:}" failed. No retries permitted until 2026-09-15 21:30:17.588559507 +0000 UTC m=+1336.188471614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz" (UID: "e4424a83-cdaf-4cd0-a403-55621bc9938d") : secret "maas-api-e2e-shared-b-w2-9aeead-serving-cert" not found
Sep 15 21:30:17.088940 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.088921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4424a83-cdaf-4cd0-a403-55621bc9938d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.091311 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.091287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.113198 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.113167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q54m\" (UniqueName: \"kubernetes.io/projected/e4424a83-cdaf-4cd0-a403-55621bc9938d-kube-api-access-5q54m\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.594082 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.594042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.597142 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.597115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:17.791064 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:17.791032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:18.098380 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:18.098353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"]
Sep 15 21:30:18.102086 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:30:18.102060    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4424a83_cdaf_4cd0_a403_55621bc9938d.slice/crio-a74524410e6567116556499a95930c135e8e96cbfbcfe91133687090bf9cb806 WatchSource:0}: Error finding container a74524410e6567116556499a95930c135e8e96cbfbcfe91133687090bf9cb806: Status 404 returned error can't find the container with id a74524410e6567116556499a95930c135e8e96cbfbcfe91133687090bf9cb806
Sep 15 21:30:18.399021 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:18.398988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz" event={"ID":"e4424a83-cdaf-4cd0-a403-55621bc9938d","Type":"ContainerStarted","Data":"a74524410e6567116556499a95930c135e8e96cbfbcfe91133687090bf9cb806"}
Sep 15 21:30:19.406879 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:19.406843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz" event={"ID":"e4424a83-cdaf-4cd0-a403-55621bc9938d","Type":"ContainerStarted","Data":"c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789"}
Sep 15 21:30:19.407346 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:19.406901    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:19.424692 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:19.424651    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz" podStartSLOduration=2.953053064 podStartE2EDuration="3.42463709s" totalImagesPullingTime="471.584026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:19.421948608 +0000 UTC m=+1338.021860729" watchObservedRunningTime="2026-09-15 21:30:19.42463709 +0000 UTC m=+1338.024549199"
Sep 15 21:30:25.418437 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:25.418360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:30:48.697551 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.697516    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"]
Sep 15 21:30:48.699253 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.699227    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-70880c-29825130-qct9z"]
Sep 15 21:30:48.732268 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.732242    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"]
Sep 15 21:30:48.732481 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.732437    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb" podUID="9ca1d149-6629-469a-a3ac-ed7e35438b85" containerName="payload-pre-processing" containerID="cri-o://75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3" gracePeriod=30
Sep 15 21:30:48.979668 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.979642    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:30:48.990787 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.990757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca1d149-6629-469a-a3ac-ed7e35438b85-plugins-config-volume\") pod \"9ca1d149-6629-469a-a3ac-ed7e35438b85\" (UID: \"9ca1d149-6629-469a-a3ac-ed7e35438b85\") "
Sep 15 21:30:48.990951 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.990820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4krk\" (UniqueName: \"kubernetes.io/projected/9ca1d149-6629-469a-a3ac-ed7e35438b85-kube-api-access-q4krk\") pod \"9ca1d149-6629-469a-a3ac-ed7e35438b85\" (UID: \"9ca1d149-6629-469a-a3ac-ed7e35438b85\") "
Sep 15 21:30:48.991508 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.991138    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca1d149-6629-469a-a3ac-ed7e35438b85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ca1d149-6629-469a-a3ac-ed7e35438b85" (UID: "9ca1d149-6629-469a-a3ac-ed7e35438b85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:48.993266 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:48.993239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca1d149-6629-469a-a3ac-ed7e35438b85-kube-api-access-q4krk" (OuterVolumeSpecName: "kube-api-access-q4krk") pod "9ca1d149-6629-469a-a3ac-ed7e35438b85" (UID: "9ca1d149-6629-469a-a3ac-ed7e35438b85"). InnerVolumeSpecName "kube-api-access-q4krk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:49.092418 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.092386    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca1d149-6629-469a-a3ac-ed7e35438b85-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:30:49.092418 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.092411    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4krk\" (UniqueName: \"kubernetes.io/projected/9ca1d149-6629-469a-a3ac-ed7e35438b85-kube-api-access-q4krk\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:30:49.576074 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.576039    2563 generic.go:356] "Generic (PLEG): container finished" podID="9ca1d149-6629-469a-a3ac-ed7e35438b85" containerID="75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3" exitCode=0
Sep 15 21:30:49.576235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.576081    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"
Sep 15 21:30:49.576235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.576118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb" event={"ID":"9ca1d149-6629-469a-a3ac-ed7e35438b85","Type":"ContainerDied","Data":"75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3"}
Sep 15 21:30:49.576235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.576171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb" event={"ID":"9ca1d149-6629-469a-a3ac-ed7e35438b85","Type":"ContainerDied","Data":"2266e12e8b9f7daadb121ac13f9166c7a6878e4f7a5b7c51a925755e9c10f623"}
Sep 15 21:30:49.576235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.576199    2563 scope.go:117] "RemoveContainer" containerID="75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3"
Sep 15 21:30:49.585367 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.585349    2563 scope.go:117] "RemoveContainer" containerID="75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3"
Sep 15 21:30:49.585650 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:30:49.585633    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3\": container with ID starting with 75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3 not found: ID does not exist" containerID="75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3"
Sep 15 21:30:49.585698 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.585659    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3"} err="failed to get container status \"75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3\": rpc error: code = NotFound desc = could not find container \"75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3\": container with ID starting with 75c13c10104ffd91fa6ad9ee056932e697e9815eb9ecd7a96ebc68e1f6649bb3 not found: ID does not exist"
Sep 15 21:30:49.614525 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.614502    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"]
Sep 15 21:30:49.617528 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.617506    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-70880c-cb4547854-blsxb"]
Sep 15 21:30:49.903884 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.903811    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f3dcb0d-bf32-4edd-8656-6dbf6e715aef" path="/var/lib/kubelet/pods/0f3dcb0d-bf32-4edd-8656-6dbf6e715aef/volumes"
Sep 15 21:30:49.904248 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:30:49.904146    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca1d149-6629-469a-a3ac-ed7e35438b85" path="/var/lib/kubelet/pods/9ca1d149-6629-469a-a3ac-ed7e35438b85/volumes"
Sep 15 21:31:01.517331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.517294    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"]
Sep 15 21:31:01.517996 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.517975    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca1d149-6629-469a-a3ac-ed7e35438b85" containerName="payload-pre-processing"
Sep 15 21:31:01.518094 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.517999    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca1d149-6629-469a-a3ac-ed7e35438b85" containerName="payload-pre-processing"
Sep 15 21:31:01.518155 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.518145    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca1d149-6629-469a-a3ac-ed7e35438b85" containerName="payload-pre-processing"
Sep 15 21:31:01.521358 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.521335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:01.523725 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.523704    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4e476d59-dockercfg-9pd2h\""
Sep 15 21:31:01.523836 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.523710    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4e476d59\""
Sep 15 21:31:01.533056 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.533036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"]
Sep 15 21:31:01.619121 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.619089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9889793-b5c3-4f23-9d0f-206c04122610-plugins-config-volume\") pod \"payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q\" (UID: \"d9889793-b5c3-4f23-9d0f-206c04122610\") " pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:01.619298 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.619197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c728\" (UniqueName: \"kubernetes.io/projected/d9889793-b5c3-4f23-9d0f-206c04122610-kube-api-access-9c728\") pod \"payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q\" (UID: \"d9889793-b5c3-4f23-9d0f-206c04122610\") " pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:01.720754 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.720723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c728\" (UniqueName: \"kubernetes.io/projected/d9889793-b5c3-4f23-9d0f-206c04122610-kube-api-access-9c728\") pod \"payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q\" (UID: \"d9889793-b5c3-4f23-9d0f-206c04122610\") " pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:01.720951 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.720789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9889793-b5c3-4f23-9d0f-206c04122610-plugins-config-volume\") pod \"payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q\" (UID: \"d9889793-b5c3-4f23-9d0f-206c04122610\") " pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:01.721482 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.721427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9889793-b5c3-4f23-9d0f-206c04122610-plugins-config-volume\") pod \"payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q\" (UID: \"d9889793-b5c3-4f23-9d0f-206c04122610\") " pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:01.729204 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.729175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c728\" (UniqueName: \"kubernetes.io/projected/d9889793-b5c3-4f23-9d0f-206c04122610-kube-api-access-9c728\") pod \"payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q\" (UID: \"d9889793-b5c3-4f23-9d0f-206c04122610\") " pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:01.835144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.835066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4e476d59-dockercfg-9pd2h\""
Sep 15 21:31:01.843013 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:01.842987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:02.031267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:02.031236    2563 scope.go:117] "RemoveContainer" containerID="b09d953202e6d776e20e9342f20b3d7034155168cdee1a0014292bb02b5d9c2e"
Sep 15 21:31:02.128624 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:02.128604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"]
Sep 15 21:31:02.130165 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:31:02.130139    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9889793_b5c3_4f23_9d0f_206c04122610.slice/crio-986b0aa912a527217a747ba29043bf2e9e0962634fd3510c273e6522b26bf29a WatchSource:0}: Error finding container 986b0aa912a527217a747ba29043bf2e9e0962634fd3510c273e6522b26bf29a: Status 404 returned error can't find the container with id 986b0aa912a527217a747ba29043bf2e9e0962634fd3510c273e6522b26bf29a
Sep 15 21:31:02.653248 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:02.653196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q" event={"ID":"d9889793-b5c3-4f23-9d0f-206c04122610","Type":"ContainerStarted","Data":"ccef2e77e567b3eb95d7b42ae8a8135749755bfee2c0a0479486a94bc46c29f0"}
Sep 15 21:31:02.653647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:02.653296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q" event={"ID":"d9889793-b5c3-4f23-9d0f-206c04122610","Type":"ContainerStarted","Data":"986b0aa912a527217a747ba29043bf2e9e0962634fd3510c273e6522b26bf29a"}
Sep 15 21:31:02.653647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:02.653326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:02.679901 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:02.679855    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q" podStartSLOduration=1.679841017 podStartE2EDuration="1.679841017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:02.67800951 +0000 UTC m=+1381.277921621" watchObservedRunningTime="2026-09-15 21:31:02.679841017 +0000 UTC m=+1381.279753126"
Sep 15 21:31:06.338224 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.338081    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"]
Sep 15 21:31:06.338820 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.338325    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9" podUID="bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" containerName="maas-api" containerID="cri-o://e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289" gracePeriod=30
Sep 15 21:31:06.400998 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.400961    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"]
Sep 15 21:31:06.401264 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.401211    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25" podUID="ff1d33a5-8be5-46b7-b59e-166055295498" containerName="payload-processing" containerID="cri-o://ea5b377b0ec5b22599e79f4c5a722939840460e10e182bb22c089ecbe44b0c3d" gracePeriod=30
Sep 15 21:31:06.661611 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.661582    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:31:06.679060 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.678985    2563 generic.go:356] "Generic (PLEG): container finished" podID="ff1d33a5-8be5-46b7-b59e-166055295498" containerID="ea5b377b0ec5b22599e79f4c5a722939840460e10e182bb22c089ecbe44b0c3d" exitCode=0
Sep 15 21:31:06.679200 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.679074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25" event={"ID":"ff1d33a5-8be5-46b7-b59e-166055295498","Type":"ContainerDied","Data":"ea5b377b0ec5b22599e79f4c5a722939840460e10e182bb22c089ecbe44b0c3d"}
Sep 15 21:31:06.679200 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.679137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25" event={"ID":"ff1d33a5-8be5-46b7-b59e-166055295498","Type":"ContainerDied","Data":"a9d481098e056004fbb06082895220709213c83ba795af2f028a08bd2e78945c"}
Sep 15 21:31:06.679200 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.679160    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9d481098e056004fbb06082895220709213c83ba795af2f028a08bd2e78945c"
Sep 15 21:31:06.682888 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.682738    2563 generic.go:356] "Generic (PLEG): container finished" podID="bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" containerID="e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289" exitCode=0
Sep 15 21:31:06.682888 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.682782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9" event={"ID":"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954","Type":"ContainerDied","Data":"e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289"}
Sep 15 21:31:06.683060 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.682906    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"
Sep 15 21:31:06.683283 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.683260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9" event={"ID":"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954","Type":"ContainerDied","Data":"b751686040ae75320a365c806e80cd48f9b1f28832a8cf2c19413aff07bebaab"}
Sep 15 21:31:06.683382 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.683316    2563 scope.go:117] "RemoveContainer" containerID="e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289"
Sep 15 21:31:06.684906 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.684835    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:31:06.712848 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.712821    2563 scope.go:117] "RemoveContainer" containerID="e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289"
Sep 15 21:31:06.713526 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:31:06.713504    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289\": container with ID starting with e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289 not found: ID does not exist" containerID="e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289"
Sep 15 21:31:06.713681 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.713580    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289"} err="failed to get container status \"e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289\": rpc error: code = NotFound desc = could not find container \"e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289\": container with ID starting with e0049b92fdf83529ceeccc5769d7bbf79642f53a99b182eff6542d862127e289 not found: ID does not exist"
Sep 15 21:31:06.763284 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67lg7\" (UniqueName: \"kubernetes.io/projected/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-kube-api-access-67lg7\") pod \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") "
Sep 15 21:31:06.763486 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg64w\" (UniqueName: \"kubernetes.io/projected/ff1d33a5-8be5-46b7-b59e-166055295498-kube-api-access-hg64w\") pod \"ff1d33a5-8be5-46b7-b59e-166055295498\" (UID: \"ff1d33a5-8be5-46b7-b59e-166055295498\") "
Sep 15 21:31:06.763486 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763332    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-metrics-certs\") pod \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") "
Sep 15 21:31:06.763486 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-openshift-service-ca\") pod \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") "
Sep 15 21:31:06.763486 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763426    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1d33a5-8be5-46b7-b59e-166055295498-plugins-config-volume\") pod \"ff1d33a5-8be5-46b7-b59e-166055295498\" (UID: \"ff1d33a5-8be5-46b7-b59e-166055295498\") "
Sep 15 21:31:06.763720 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763513    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls\") pod \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\" (UID: \"bf1d0a7b-7e3c-4fb2-ace6-a180dff25954\") "
Sep 15 21:31:06.763838 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" (UID: "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:06.763930 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763911    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:31:06.763990 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.763925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1d33a5-8be5-46b7-b59e-166055295498-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff1d33a5-8be5-46b7-b59e-166055295498" (UID: "ff1d33a5-8be5-46b7-b59e-166055295498"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:06.765830 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.765797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" (UID: "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:06.765939 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.765890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-kube-api-access-67lg7" (OuterVolumeSpecName: "kube-api-access-67lg7") pod "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" (UID: "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954"). InnerVolumeSpecName "kube-api-access-67lg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:06.766092 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.766072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1d33a5-8be5-46b7-b59e-166055295498-kube-api-access-hg64w" (OuterVolumeSpecName: "kube-api-access-hg64w") pod "ff1d33a5-8be5-46b7-b59e-166055295498" (UID: "ff1d33a5-8be5-46b7-b59e-166055295498"). InnerVolumeSpecName "kube-api-access-hg64w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:06.766174 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.766111    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" (UID: "bf1d0a7b-7e3c-4fb2-ace6-a180dff25954"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:06.865468 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.865370    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1d33a5-8be5-46b7-b59e-166055295498-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:31:06.865468 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.865397    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:31:06.865468 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.865405    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lg7\" (UniqueName: \"kubernetes.io/projected/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-kube-api-access-67lg7\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:31:06.865468 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.865415    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg64w\" (UniqueName: \"kubernetes.io/projected/ff1d33a5-8be5-46b7-b59e-166055295498-kube-api-access-hg64w\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:31:06.865468 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:06.865424    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:31:07.005232 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:07.005202    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"]
Sep 15 21:31:07.011427 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:07.011403    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-779ace-768dc97756-wxhh9"]
Sep 15 21:31:07.688977 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:07.688904    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"
Sep 15 21:31:07.712895 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:07.712860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"]
Sep 15 21:31:07.718617 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:07.718595    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-779ace-5b4bfc8c5-9sn25"]
Sep 15 21:31:07.902391 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:07.902359    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" path="/var/lib/kubelet/pods/bf1d0a7b-7e3c-4fb2-ace6-a180dff25954/volumes"
Sep 15 21:31:07.902783 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:07.902769    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff1d33a5-8be5-46b7-b59e-166055295498" path="/var/lib/kubelet/pods/ff1d33a5-8be5-46b7-b59e-166055295498/volumes"
Sep 15 21:31:13.661528 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:13.661492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:31:57.763532 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.763255    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"]
Sep 15 21:31:57.763910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.763876    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1d33a5-8be5-46b7-b59e-166055295498" containerName="payload-processing"
Sep 15 21:31:57.763910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.763892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1d33a5-8be5-46b7-b59e-166055295498" containerName="payload-processing"
Sep 15 21:31:57.763910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.763905    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" containerName="maas-api"
Sep 15 21:31:57.763910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.763910    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" containerName="maas-api"
Sep 15 21:31:57.764044 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.763981    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1d33a5-8be5-46b7-b59e-166055295498" containerName="payload-processing"
Sep 15 21:31:57.764044 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.763990    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf1d0a7b-7e3c-4fb2-ace6-a180dff25954" containerName="maas-api"
Sep 15 21:31:57.767549 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.767532    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.769775 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.769753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-3c4dc8-serving-cert\""
Sep 15 21:31:57.784802 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.784777    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"]
Sep 15 21:31:57.863303 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.863267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.863303 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.863308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.863565 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.863418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96950d4-cc38-4001-ac6f-85b9aa344755-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.863565 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.863547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9vb9\" (UniqueName: \"kubernetes.io/projected/e96950d4-cc38-4001-ac6f-85b9aa344755-kube-api-access-z9vb9\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.964418 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.964385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96950d4-cc38-4001-ac6f-85b9aa344755-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.964647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.964434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9vb9\" (UniqueName: \"kubernetes.io/projected/e96950d4-cc38-4001-ac6f-85b9aa344755-kube-api-access-z9vb9\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.964647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.964516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.964647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.964538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.964809 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:31:57.964661    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-serving-cert: secret "maas-api-e2e-shared-a-w3-3c4dc8-serving-cert" not found
Sep 15 21:31:57.964809 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:31:57.964743    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls podName:e96950d4-cc38-4001-ac6f-85b9aa344755 nodeName:}" failed. No retries permitted until 2026-09-15 21:31:58.464721436 +0000 UTC m=+1437.064633525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls") pod "maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2" (UID: "e96950d4-cc38-4001-ac6f-85b9aa344755") : secret "maas-api-e2e-shared-a-w3-3c4dc8-serving-cert" not found
Sep 15 21:31:57.965089 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.965063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96950d4-cc38-4001-ac6f-85b9aa344755-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.967188 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.967160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:57.973117 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:57.973094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9vb9\" (UniqueName: \"kubernetes.io/projected/e96950d4-cc38-4001-ac6f-85b9aa344755-kube-api-access-z9vb9\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:58.470168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:58.470135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:58.470385 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:31:58.470309    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-serving-cert: secret "maas-api-e2e-shared-a-w3-3c4dc8-serving-cert" not found
Sep 15 21:31:58.470440 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:31:58.470392    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls podName:e96950d4-cc38-4001-ac6f-85b9aa344755 nodeName:}" failed. No retries permitted until 2026-09-15 21:31:59.470375583 +0000 UTC m=+1438.070287671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls") pod "maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2" (UID: "e96950d4-cc38-4001-ac6f-85b9aa344755") : secret "maas-api-e2e-shared-a-w3-3c4dc8-serving-cert" not found
Sep 15 21:31:59.481546 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:59.481511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:59.484478 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:59.484424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:59.578725 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:59.578688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:31:59.853861 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:59.853834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"]
Sep 15 21:31:59.856566 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:31:59.856540    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96950d4_cc38_4001_ac6f_85b9aa344755.slice/crio-cef9219d1969e0454d09aeb00c08dad244cc326b9d32a6b177bfc1cfcf6a9feb WatchSource:0}: Error finding container cef9219d1969e0454d09aeb00c08dad244cc326b9d32a6b177bfc1cfcf6a9feb: Status 404 returned error can't find the container with id cef9219d1969e0454d09aeb00c08dad244cc326b9d32a6b177bfc1cfcf6a9feb
Sep 15 21:31:59.995672 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:31:59.995638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2" event={"ID":"e96950d4-cc38-4001-ac6f-85b9aa344755","Type":"ContainerStarted","Data":"cef9219d1969e0454d09aeb00c08dad244cc326b9d32a6b177bfc1cfcf6a9feb"}
Sep 15 21:32:02.055909 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:02.055877    2563 scope.go:117] "RemoveContainer" containerID="ea5b377b0ec5b22599e79f4c5a722939840460e10e182bb22c089ecbe44b0c3d"
Sep 15 21:32:03.014194 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:03.014142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2" event={"ID":"e96950d4-cc38-4001-ac6f-85b9aa344755","Type":"ContainerStarted","Data":"9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f"}
Sep 15 21:32:03.014380 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:03.014310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:32:03.039235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:03.039189    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2" podStartSLOduration=3.7521497249999998 podStartE2EDuration="6.039174295s" totalImagesPullingTime="2.28702457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:03.0368196 +0000 UTC m=+1441.636731710" watchObservedRunningTime="2026-09-15 21:32:03.039174295 +0000 UTC m=+1441.639086405"
Sep 15 21:32:09.025075 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:09.025042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:32:32.652608 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.652571    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"]
Sep 15 21:32:32.656590 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.656565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.658652 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.658631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7c57b6-serving-cert\""
Sep 15 21:32:32.668412 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.668388    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"]
Sep 15 21:32:32.803817 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.803781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.804002 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.803851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tzn\" (UniqueName: \"kubernetes.io/projected/b17b4bf0-44d0-418d-9537-02e1690fb98b-kube-api-access-l5tzn\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.804002 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.803918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.804109 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.803998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17b4bf0-44d0-418d-9537-02e1690fb98b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.905295 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.905227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17b4bf0-44d0-418d-9537-02e1690fb98b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.905295 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.905263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.905545 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.905316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tzn\" (UniqueName: \"kubernetes.io/projected/b17b4bf0-44d0-418d-9537-02e1690fb98b-kube-api-access-l5tzn\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.905545 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.905380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.905545 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:32:32.905482    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-serving-cert: secret "maas-api-e2e-shared-b-w3-7c57b6-serving-cert" not found
Sep 15 21:32:32.905718 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:32:32.905567    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls podName:b17b4bf0-44d0-418d-9537-02e1690fb98b nodeName:}" failed. No retries permitted until 2026-09-15 21:32:33.405544675 +0000 UTC m=+1472.005456779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7" (UID: "b17b4bf0-44d0-418d-9537-02e1690fb98b") : secret "maas-api-e2e-shared-b-w3-7c57b6-serving-cert" not found
Sep 15 21:32:32.906016 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.905999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17b4bf0-44d0-418d-9537-02e1690fb98b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.907967 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.907946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:32.916890 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:32.916868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tzn\" (UniqueName: \"kubernetes.io/projected/b17b4bf0-44d0-418d-9537-02e1690fb98b-kube-api-access-l5tzn\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:33.312786 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.312750    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"]
Sep 15 21:32:33.316679 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.316663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:33.319407 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.319383    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7c57b6\""
Sep 15 21:32:33.319544 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.319504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7c57b6-dockercfg-dvmwb\""
Sep 15 21:32:33.329069 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.329048    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"]
Sep 15 21:32:33.411591 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.411557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:33.411591 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.411598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301211d7-dedf-4cbe-a2e2-17048319187c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj\" (UID: \"301211d7-dedf-4cbe-a2e2-17048319187c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:33.411850 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.411619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8kg\" (UniqueName: \"kubernetes.io/projected/301211d7-dedf-4cbe-a2e2-17048319187c-kube-api-access-bw8kg\") pod \"payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj\" (UID: \"301211d7-dedf-4cbe-a2e2-17048319187c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:33.414972 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.414940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:33.513327 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.513278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301211d7-dedf-4cbe-a2e2-17048319187c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj\" (UID: \"301211d7-dedf-4cbe-a2e2-17048319187c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:33.513327 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.513322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8kg\" (UniqueName: \"kubernetes.io/projected/301211d7-dedf-4cbe-a2e2-17048319187c-kube-api-access-bw8kg\") pod \"payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj\" (UID: \"301211d7-dedf-4cbe-a2e2-17048319187c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:33.514568 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.514539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301211d7-dedf-4cbe-a2e2-17048319187c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj\" (UID: \"301211d7-dedf-4cbe-a2e2-17048319187c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:33.522773 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.522744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8kg\" (UniqueName: \"kubernetes.io/projected/301211d7-dedf-4cbe-a2e2-17048319187c-kube-api-access-bw8kg\") pod \"payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj\" (UID: \"301211d7-dedf-4cbe-a2e2-17048319187c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:33.568664 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.568592    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:33.628737 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.628514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:33.842457 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.842424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"]
Sep 15 21:32:33.844359 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:32:33.844334    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17b4bf0_44d0_418d_9537_02e1690fb98b.slice/crio-8b580be3a4ded41a14e5201a54848f671c77c5e3db58f7dd29102438dca7710c WatchSource:0}: Error finding container 8b580be3a4ded41a14e5201a54848f671c77c5e3db58f7dd29102438dca7710c: Status 404 returned error can't find the container with id 8b580be3a4ded41a14e5201a54848f671c77c5e3db58f7dd29102438dca7710c
Sep 15 21:32:33.918540 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:33.918512    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"]
Sep 15 21:32:33.919779 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:32:33.919754    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301211d7_dedf_4cbe_a2e2_17048319187c.slice/crio-a2707bcc76bc1c4daf7fca5431bf4631db47989f90c2aec00269e9e4b1004745 WatchSource:0}: Error finding container a2707bcc76bc1c4daf7fca5431bf4631db47989f90c2aec00269e9e4b1004745: Status 404 returned error can't find the container with id a2707bcc76bc1c4daf7fca5431bf4631db47989f90c2aec00269e9e4b1004745
Sep 15 21:32:34.192694 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:34.192595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7" event={"ID":"b17b4bf0-44d0-418d-9537-02e1690fb98b","Type":"ContainerStarted","Data":"8b580be3a4ded41a14e5201a54848f671c77c5e3db58f7dd29102438dca7710c"}
Sep 15 21:32:34.194637 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:34.194604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj" event={"ID":"301211d7-dedf-4cbe-a2e2-17048319187c","Type":"ContainerStarted","Data":"46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013"}
Sep 15 21:32:34.194784 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:34.194652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj" event={"ID":"301211d7-dedf-4cbe-a2e2-17048319187c","Type":"ContainerStarted","Data":"a2707bcc76bc1c4daf7fca5431bf4631db47989f90c2aec00269e9e4b1004745"}
Sep 15 21:32:34.194784 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:34.194674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:32:34.212127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:34.212087    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj" podStartSLOduration=1.212073092 podStartE2EDuration="1.212073092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:34.208824115 +0000 UTC m=+1472.808736224" watchObservedRunningTime="2026-09-15 21:32:34.212073092 +0000 UTC m=+1472.811985202"
Sep 15 21:32:35.202682 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:35.202637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7" event={"ID":"b17b4bf0-44d0-418d-9537-02e1690fb98b","Type":"ContainerStarted","Data":"bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1"}
Sep 15 21:32:35.224271 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:35.224218    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7" podStartSLOduration=2.540180925 podStartE2EDuration="3.224203366s" totalImagesPullingTime="684.022441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:35.220805871 +0000 UTC m=+1473.820717994" watchObservedRunningTime="2026-09-15 21:32:35.224203366 +0000 UTC m=+1473.824115475"
Sep 15 21:32:36.208040 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:36.208010    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:42.219464 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:42.219417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:32:45.205436 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:32:45.205406    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:34:19.604343 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:19.604307    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"]
Sep 15 21:34:19.607330 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:19.604568    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q" podUID="d9889793-b5c3-4f23-9d0f-206c04122610" containerName="payload-processing" containerID="cri-o://ccef2e77e567b3eb95d7b42ae8a8135749755bfee2c0a0479486a94bc46c29f0" gracePeriod=30
Sep 15 21:34:19.811356 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:19.811236    2563 generic.go:356] "Generic (PLEG): container finished" podID="d9889793-b5c3-4f23-9d0f-206c04122610" containerID="ccef2e77e567b3eb95d7b42ae8a8135749755bfee2c0a0479486a94bc46c29f0" exitCode=0
Sep 15 21:34:19.811356 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:19.811294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q" event={"ID":"d9889793-b5c3-4f23-9d0f-206c04122610","Type":"ContainerDied","Data":"ccef2e77e567b3eb95d7b42ae8a8135749755bfee2c0a0479486a94bc46c29f0"}
Sep 15 21:34:19.874370 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:19.874347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:34:20.046613 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.046582    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9889793-b5c3-4f23-9d0f-206c04122610-plugins-config-volume\") pod \"d9889793-b5c3-4f23-9d0f-206c04122610\" (UID: \"d9889793-b5c3-4f23-9d0f-206c04122610\") "
Sep 15 21:34:20.046781 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.046643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c728\" (UniqueName: \"kubernetes.io/projected/d9889793-b5c3-4f23-9d0f-206c04122610-kube-api-access-9c728\") pod \"d9889793-b5c3-4f23-9d0f-206c04122610\" (UID: \"d9889793-b5c3-4f23-9d0f-206c04122610\") "
Sep 15 21:34:20.047013 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.046987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9889793-b5c3-4f23-9d0f-206c04122610-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9889793-b5c3-4f23-9d0f-206c04122610" (UID: "d9889793-b5c3-4f23-9d0f-206c04122610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:34:20.048915 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.048883    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9889793-b5c3-4f23-9d0f-206c04122610-kube-api-access-9c728" (OuterVolumeSpecName: "kube-api-access-9c728") pod "d9889793-b5c3-4f23-9d0f-206c04122610" (UID: "d9889793-b5c3-4f23-9d0f-206c04122610"). InnerVolumeSpecName "kube-api-access-9c728". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:20.148207 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.148134    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9889793-b5c3-4f23-9d0f-206c04122610-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:34:20.148207 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.148160    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c728\" (UniqueName: \"kubernetes.io/projected/d9889793-b5c3-4f23-9d0f-206c04122610-kube-api-access-9c728\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:34:20.818031 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.817992    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"
Sep 15 21:34:20.818441 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.818029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q" event={"ID":"d9889793-b5c3-4f23-9d0f-206c04122610","Type":"ContainerDied","Data":"986b0aa912a527217a747ba29043bf2e9e0962634fd3510c273e6522b26bf29a"}
Sep 15 21:34:20.818441 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.818100    2563 scope.go:117] "RemoveContainer" containerID="ccef2e77e567b3eb95d7b42ae8a8135749755bfee2c0a0479486a94bc46c29f0"
Sep 15 21:34:20.843561 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.843531    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"]
Sep 15 21:34:20.846096 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:20.846075    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4e476d59-6bff79cbbf-mqp4q"]
Sep 15 21:34:21.902331 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:34:21.902299    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9889793-b5c3-4f23-9d0f-206c04122610" path="/var/lib/kubelet/pods/d9889793-b5c3-4f23-9d0f-206c04122610/volumes"
Sep 15 21:36:02.114495 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:02.114467    2563 scope.go:117] "RemoveContainer" containerID="39143ebf0905d8184f0308de8aef8a34d91b14f9a93e19b2be0b68dad1448b2b"
Sep 15 21:36:22.365085 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.365053    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"]
Sep 15 21:36:22.366524 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.366498    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9889793-b5c3-4f23-9d0f-206c04122610" containerName="payload-processing"
Sep 15 21:36:22.366524 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.366525    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9889793-b5c3-4f23-9d0f-206c04122610" containerName="payload-processing"
Sep 15 21:36:22.366714 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.366651    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9889793-b5c3-4f23-9d0f-206c04122610" containerName="payload-processing"
Sep 15 21:36:22.371949 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.371930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.373965 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.373945    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0753ad-serving-cert\""
Sep 15 21:36:22.383018 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.382835    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"]
Sep 15 21:36:22.519071 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.519040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwbc\" (UniqueName: \"kubernetes.io/projected/d886e7db-d050-416c-b971-7eea8ff7173c-kube-api-access-hwwbc\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.519259 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.519078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886e7db-d050-416c-b971-7eea8ff7173c-openshift-service-ca\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.519259 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.519101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-metrics-certs\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.519340 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.519261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-maas-api-tls\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.620582 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.620484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-maas-api-tls\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.620582 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.620534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwbc\" (UniqueName: \"kubernetes.io/projected/d886e7db-d050-416c-b971-7eea8ff7173c-kube-api-access-hwwbc\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.620582 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.620558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886e7db-d050-416c-b971-7eea8ff7173c-openshift-service-ca\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.620876 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.620589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-metrics-certs\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.621374 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.621345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886e7db-d050-416c-b971-7eea8ff7173c-openshift-service-ca\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.623185 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.623164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-metrics-certs\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.623281 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.623204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-maas-api-tls\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.628704 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.628681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwbc\" (UniqueName: \"kubernetes.io/projected/d886e7db-d050-416c-b971-7eea8ff7173c-kube-api-access-hwwbc\") pod \"maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.686966 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.686932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:22.971155 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.971132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"]
Sep 15 21:36:22.973182 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:36:22.973156    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd886e7db_d050_416c_b971_7eea8ff7173c.slice/crio-804625e6ea6278cd95eb7241037c22301892eaec06a8679438f8b710b7dc89fe WatchSource:0}: Error finding container 804625e6ea6278cd95eb7241037c22301892eaec06a8679438f8b710b7dc89fe: Status 404 returned error can't find the container with id 804625e6ea6278cd95eb7241037c22301892eaec06a8679438f8b710b7dc89fe
Sep 15 21:36:22.974749 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:22.974731    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:36:23.072819 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.072787    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"]
Sep 15 21:36:23.077913 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.077897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:23.080545 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.080516    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0753ad-dockercfg-rtqdx\""
Sep 15 21:36:23.080870 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.080851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0753ad\""
Sep 15 21:36:23.089128 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.089084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"]
Sep 15 21:36:23.226879 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.226797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffh46\" (UniqueName: \"kubernetes.io/projected/bfce212f-de75-488c-b573-cacfa5bb1553-kube-api-access-ffh46\") pod \"payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc\" (UID: \"bfce212f-de75-488c-b573-cacfa5bb1553\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:23.226879 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.226875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfce212f-de75-488c-b573-cacfa5bb1553-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc\" (UID: \"bfce212f-de75-488c-b573-cacfa5bb1553\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:23.327831 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.327800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffh46\" (UniqueName: \"kubernetes.io/projected/bfce212f-de75-488c-b573-cacfa5bb1553-kube-api-access-ffh46\") pod \"payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc\" (UID: \"bfce212f-de75-488c-b573-cacfa5bb1553\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:23.328049 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.327889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfce212f-de75-488c-b573-cacfa5bb1553-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc\" (UID: \"bfce212f-de75-488c-b573-cacfa5bb1553\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:23.328575 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.328557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfce212f-de75-488c-b573-cacfa5bb1553-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc\" (UID: \"bfce212f-de75-488c-b573-cacfa5bb1553\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:23.335977 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.335953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffh46\" (UniqueName: \"kubernetes.io/projected/bfce212f-de75-488c-b573-cacfa5bb1553-kube-api-access-ffh46\") pod \"payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc\" (UID: \"bfce212f-de75-488c-b573-cacfa5bb1553\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:23.390103 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.390075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:23.510707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.510622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz" event={"ID":"d886e7db-d050-416c-b971-7eea8ff7173c","Type":"ContainerStarted","Data":"804625e6ea6278cd95eb7241037c22301892eaec06a8679438f8b710b7dc89fe"}
Sep 15 21:36:23.668723 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:23.668699    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"]
Sep 15 21:36:23.670977 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:36:23.670945    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfce212f_de75_488c_b573_cacfa5bb1553.slice/crio-f734866262006a40ef671217c88146ef3dace012dee150d18f9db52cbb3a8e2d WatchSource:0}: Error finding container f734866262006a40ef671217c88146ef3dace012dee150d18f9db52cbb3a8e2d: Status 404 returned error can't find the container with id f734866262006a40ef671217c88146ef3dace012dee150d18f9db52cbb3a8e2d
Sep 15 21:36:24.518057 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:24.518026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc" event={"ID":"bfce212f-de75-488c-b573-cacfa5bb1553","Type":"ContainerStarted","Data":"fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b"}
Sep 15 21:36:24.518524 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:24.518078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc" event={"ID":"bfce212f-de75-488c-b573-cacfa5bb1553","Type":"ContainerStarted","Data":"f734866262006a40ef671217c88146ef3dace012dee150d18f9db52cbb3a8e2d"}
Sep 15 21:36:24.518524 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:24.518129    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:24.519827 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:24.519805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz" event={"ID":"d886e7db-d050-416c-b971-7eea8ff7173c","Type":"ContainerStarted","Data":"19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3"}
Sep 15 21:36:24.519957 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:24.519943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:24.540113 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:24.536197    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc" podStartSLOduration=1.536178537 podStartE2EDuration="1.536178537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:24.532406079 +0000 UTC m=+1703.132318189" watchObservedRunningTime="2026-09-15 21:36:24.536178537 +0000 UTC m=+1703.136090649"
Sep 15 21:36:24.550235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:24.550177    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz" podStartSLOduration=2.003304461 podStartE2EDuration="2.550163829s" totalImagesPullingTime="546.859368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:24.548617748 +0000 UTC m=+1703.148529859" watchObservedRunningTime="2026-09-15 21:36:24.550163829 +0000 UTC m=+1703.150075939"
Sep 15 21:36:30.530668 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:30.530634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:36:35.528062 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:35.528026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:36:39.942302 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:39.942265    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"]
Sep 15 21:36:48.872173 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:48.872136    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"]
Sep 15 21:36:48.884130 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:48.884094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:48.887778 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:48.887520    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-30e32e-serving-cert\""
Sep 15 21:36:48.898890 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:48.898860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"]
Sep 15 21:36:48.984700 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:48.984663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-metrics-certs\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:48.984883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:48.984757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:48.984883 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:48.984798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f18b9b0-26b0-4459-8582-a75bb520cc48-openshift-service-ca\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:48.984979 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:48.984884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrf9\" (UniqueName: \"kubernetes.io/projected/3f18b9b0-26b0-4459-8582-a75bb520cc48-kube-api-access-pjrf9\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.086496 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.086440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f18b9b0-26b0-4459-8582-a75bb520cc48-openshift-service-ca\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.086834 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.086574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrf9\" (UniqueName: \"kubernetes.io/projected/3f18b9b0-26b0-4459-8582-a75bb520cc48-kube-api-access-pjrf9\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.086834 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.086664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-metrics-certs\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.086834 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.086764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.087024 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:36:49.086913    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-serving-cert: secret "maas-api-e2e-api-b-30e32e-serving-cert" not found
Sep 15 21:36:49.087024 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:36:49.086973    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls podName:3f18b9b0-26b0-4459-8582-a75bb520cc48 nodeName:}" failed. No retries permitted until 2026-09-15 21:36:49.586957222 +0000 UTC m=+1728.186869310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls") pod "maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg" (UID: "3f18b9b0-26b0-4459-8582-a75bb520cc48") : secret "maas-api-e2e-api-b-30e32e-serving-cert" not found
Sep 15 21:36:49.087309 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.087260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f18b9b0-26b0-4459-8582-a75bb520cc48-openshift-service-ca\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.089558 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.089535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-metrics-certs\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.095731 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.095704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrf9\" (UniqueName: \"kubernetes.io/projected/3f18b9b0-26b0-4459-8582-a75bb520cc48-kube-api-access-pjrf9\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.592038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.591996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.594748 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.594720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls\") pod \"maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.804435 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.804396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:49.834137 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.834108    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"]
Sep 15 21:36:49.838273 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.838251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:49.840511 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.840482    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-30e32e\""
Sep 15 21:36:49.840644 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.840549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-30e32e-dockercfg-9xv48\""
Sep 15 21:36:49.848202 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.848178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"]
Sep 15 21:36:49.895180 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.895110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69c8d4-3435-471b-8328-3aa1bfc4106a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj\" (UID: \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\") " pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:49.895592 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.895297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbkdk\" (UniqueName: \"kubernetes.io/projected/3f69c8d4-3435-471b-8328-3aa1bfc4106a-kube-api-access-fbkdk\") pod \"payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj\" (UID: \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\") " pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:49.997221 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.996874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbkdk\" (UniqueName: \"kubernetes.io/projected/3f69c8d4-3435-471b-8328-3aa1bfc4106a-kube-api-access-fbkdk\") pod \"payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj\" (UID: \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\") " pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:49.997221 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.996958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69c8d4-3435-471b-8328-3aa1bfc4106a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj\" (UID: \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\") " pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:49.997709 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:49.997689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69c8d4-3435-471b-8328-3aa1bfc4106a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj\" (UID: \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\") " pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:50.007717 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.007687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbkdk\" (UniqueName: \"kubernetes.io/projected/3f69c8d4-3435-471b-8328-3aa1bfc4106a-kube-api-access-fbkdk\") pod \"payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj\" (UID: \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\") " pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:50.087517 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.087495    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"]
Sep 15 21:36:50.089895 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:36:50.089856    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f18b9b0_26b0_4459_8582_a75bb520cc48.slice/crio-f0d18b32b96e10efc347c64cba9b45a434139910fc70bbf53c05fd6efe74fd29 WatchSource:0}: Error finding container f0d18b32b96e10efc347c64cba9b45a434139910fc70bbf53c05fd6efe74fd29: Status 404 returned error can't find the container with id f0d18b32b96e10efc347c64cba9b45a434139910fc70bbf53c05fd6efe74fd29
Sep 15 21:36:50.151563 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.151490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:50.431526 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.431491    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"]
Sep 15 21:36:50.432845 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:36:50.432823    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f69c8d4_3435_471b_8328_3aa1bfc4106a.slice/crio-b155bc23de72bc025fdfa0bf5900d55e7c8bb8c92be7ec30e52628f265c39030 WatchSource:0}: Error finding container b155bc23de72bc025fdfa0bf5900d55e7c8bb8c92be7ec30e52628f265c39030: Status 404 returned error can't find the container with id b155bc23de72bc025fdfa0bf5900d55e7c8bb8c92be7ec30e52628f265c39030
Sep 15 21:36:50.675397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.675358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj" event={"ID":"3f69c8d4-3435-471b-8328-3aa1bfc4106a","Type":"ContainerStarted","Data":"4499f3bdf6d07cac864b41e206c0f221be3d53a3875d924789426bb73c4b07cc"}
Sep 15 21:36:50.675594 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.675416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj" event={"ID":"3f69c8d4-3435-471b-8328-3aa1bfc4106a","Type":"ContainerStarted","Data":"b155bc23de72bc025fdfa0bf5900d55e7c8bb8c92be7ec30e52628f265c39030"}
Sep 15 21:36:50.675594 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.675476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:36:50.676812 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.676781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg" event={"ID":"3f18b9b0-26b0-4459-8582-a75bb520cc48","Type":"ContainerStarted","Data":"f0d18b32b96e10efc347c64cba9b45a434139910fc70bbf53c05fd6efe74fd29"}
Sep 15 21:36:50.693064 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:50.692979    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj" podStartSLOduration=1.692961259 podStartE2EDuration="1.692961259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:50.692383071 +0000 UTC m=+1729.292295182" watchObservedRunningTime="2026-09-15 21:36:50.692961259 +0000 UTC m=+1729.292873372"
Sep 15 21:36:51.685208 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:51.685145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg" event={"ID":"3f18b9b0-26b0-4459-8582-a75bb520cc48","Type":"ContainerStarted","Data":"eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4"}
Sep 15 21:36:51.704020 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:51.703965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg" podStartSLOduration=3.05843655 podStartE2EDuration="3.703942832s" totalImagesPullingTime="645.506282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:51.702078725 +0000 UTC m=+1730.301990836" watchObservedRunningTime="2026-09-15 21:36:51.703942832 +0000 UTC m=+1730.303854946"
Sep 15 21:36:52.690615 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:52.690573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:54.963471 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:54.963388    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc" podUID="4186d9f9-39af-418d-8c62-26d8a81bb6fa" containerName="main" containerID="cri-o://60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b" gracePeriod=45
Sep 15 21:36:55.224752 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.224693    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:36:55.353312 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.353272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tmp-dir\") pod \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") "
Sep 15 21:36:55.353515 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.353365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb9dx\" (UniqueName: \"kubernetes.io/projected/4186d9f9-39af-418d-8c62-26d8a81bb6fa-kube-api-access-wb9dx\") pod \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") "
Sep 15 21:36:55.353515 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.353396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-dshm\") pod \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") "
Sep 15 21:36:55.353515 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.353422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-model-cache\") pod \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") "
Sep 15 21:36:55.353688 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.353588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4186d9f9-39af-418d-8c62-26d8a81bb6fa" (UID: "4186d9f9-39af-418d-8c62-26d8a81bb6fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.353688 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.353598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tls-certs\") pod \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") "
Sep 15 21:36:55.353796 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.353692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-home\") pod \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\" (UID: \"4186d9f9-39af-418d-8c62-26d8a81bb6fa\") "
Sep 15 21:36:55.354047 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.354017    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "4186d9f9-39af-418d-8c62-26d8a81bb6fa" (UID: "4186d9f9-39af-418d-8c62-26d8a81bb6fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.354047 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.354037    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-home" (OuterVolumeSpecName: "home") pod "4186d9f9-39af-418d-8c62-26d8a81bb6fa" (UID: "4186d9f9-39af-418d-8c62-26d8a81bb6fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.354254 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.354137    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-home\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.354254 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.354160    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tmp-dir\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.354254 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.354173    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-model-cache\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.356350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.356302    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4186d9f9-39af-418d-8c62-26d8a81bb6fa" (UID: "4186d9f9-39af-418d-8c62-26d8a81bb6fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:55.356519 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.356417    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-dshm" (OuterVolumeSpecName: "dshm") pod "4186d9f9-39af-418d-8c62-26d8a81bb6fa" (UID: "4186d9f9-39af-418d-8c62-26d8a81bb6fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.356674 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.356644    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4186d9f9-39af-418d-8c62-26d8a81bb6fa-kube-api-access-wb9dx" (OuterVolumeSpecName: "kube-api-access-wb9dx") pod "4186d9f9-39af-418d-8c62-26d8a81bb6fa" (UID: "4186d9f9-39af-418d-8c62-26d8a81bb6fa"). InnerVolumeSpecName "kube-api-access-wb9dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:55.455603 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.455505    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb9dx\" (UniqueName: \"kubernetes.io/projected/4186d9f9-39af-418d-8c62-26d8a81bb6fa-kube-api-access-wb9dx\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.455603 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.455533    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4186d9f9-39af-418d-8c62-26d8a81bb6fa-dshm\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.455603 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.455543    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4186d9f9-39af-418d-8c62-26d8a81bb6fa-tls-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.713609 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.713574    2563 generic.go:356] "Generic (PLEG): container finished" podID="4186d9f9-39af-418d-8c62-26d8a81bb6fa" containerID="60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b" exitCode=0
Sep 15 21:36:55.713755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.713615    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"
Sep 15 21:36:55.713755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.713660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc" event={"ID":"4186d9f9-39af-418d-8c62-26d8a81bb6fa","Type":"ContainerDied","Data":"60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b"}
Sep 15 21:36:55.713755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.713710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc" event={"ID":"4186d9f9-39af-418d-8c62-26d8a81bb6fa","Type":"ContainerDied","Data":"7620a5a1eb7e447957278ce72f06c3b17b338cdddc7a9adacbd33c7b8c08157a"}
Sep 15 21:36:55.713755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.713732    2563 scope.go:117] "RemoveContainer" containerID="60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b"
Sep 15 21:36:55.724205 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.724185    2563 scope.go:117] "RemoveContainer" containerID="60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b"
Sep 15 21:36:55.724484 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:36:55.724462    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b\": container with ID starting with 60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b not found: ID does not exist" containerID="60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b"
Sep 15 21:36:55.724541 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.724494    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b"} err="failed to get container status \"60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b\": rpc error: code = NotFound desc = could not find container \"60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b\": container with ID starting with 60c0b55a1ec9b407da7a7c8f4913bb58650ab86967eff42c4f19f8ad9ee6a72b not found: ID does not exist"
Sep 15 21:36:55.753796 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.753767    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"]
Sep 15 21:36:55.761542 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.761518    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-w0-a276f9-kserve-77bc4668cb-zcgfc"]
Sep 15 21:36:55.903052 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:55.903019    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4186d9f9-39af-418d-8c62-26d8a81bb6fa" path="/var/lib/kubelet/pods/4186d9f9-39af-418d-8c62-26d8a81bb6fa/volumes"
Sep 15 21:36:57.345877 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.345842    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"]
Sep 15 21:36:57.346358 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.346081    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr" podUID="7727f857-7018-4cd0-ac20-6ea0917cf48c" containerName="payload-pre-processing" containerID="cri-o://7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857" gracePeriod=30
Sep 15 21:36:57.639110 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.638925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:36:57.728945 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.728908    2563 generic.go:356] "Generic (PLEG): container finished" podID="7727f857-7018-4cd0-ac20-6ea0917cf48c" containerID="7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857" exitCode=0
Sep 15 21:36:57.729133 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.728963    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"
Sep 15 21:36:57.729133 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.728991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr" event={"ID":"7727f857-7018-4cd0-ac20-6ea0917cf48c","Type":"ContainerDied","Data":"7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857"}
Sep 15 21:36:57.729133 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.729066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr" event={"ID":"7727f857-7018-4cd0-ac20-6ea0917cf48c","Type":"ContainerDied","Data":"031b2d5d993e10fc1255e35951d0cc3e8178ecddf62b22cd9de2cee2d694dbdb"}
Sep 15 21:36:57.729133 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.729089    2563 scope.go:117] "RemoveContainer" containerID="7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857"
Sep 15 21:36:57.738326 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.738309    2563 scope.go:117] "RemoveContainer" containerID="7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857"
Sep 15 21:36:57.738615 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:36:57.738596    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857\": container with ID starting with 7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857 not found: ID does not exist" containerID="7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857"
Sep 15 21:36:57.738703 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.738623    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857"} err="failed to get container status \"7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857\": rpc error: code = NotFound desc = could not find container \"7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857\": container with ID starting with 7105eac163937d7da5a89cc76e94498cee4078e6c7bcdc86ab1d4383886e8857 not found: ID does not exist"
Sep 15 21:36:57.778243 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.778217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7727f857-7018-4cd0-ac20-6ea0917cf48c-plugins-config-volume\") pod \"7727f857-7018-4cd0-ac20-6ea0917cf48c\" (UID: \"7727f857-7018-4cd0-ac20-6ea0917cf48c\") "
Sep 15 21:36:57.778368 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.778256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df5br\" (UniqueName: \"kubernetes.io/projected/7727f857-7018-4cd0-ac20-6ea0917cf48c-kube-api-access-df5br\") pod \"7727f857-7018-4cd0-ac20-6ea0917cf48c\" (UID: \"7727f857-7018-4cd0-ac20-6ea0917cf48c\") "
Sep 15 21:36:57.778609 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.778590    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7727f857-7018-4cd0-ac20-6ea0917cf48c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7727f857-7018-4cd0-ac20-6ea0917cf48c" (UID: "7727f857-7018-4cd0-ac20-6ea0917cf48c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:57.780523 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.780505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7727f857-7018-4cd0-ac20-6ea0917cf48c-kube-api-access-df5br" (OuterVolumeSpecName: "kube-api-access-df5br") pod "7727f857-7018-4cd0-ac20-6ea0917cf48c" (UID: "7727f857-7018-4cd0-ac20-6ea0917cf48c"). InnerVolumeSpecName "kube-api-access-df5br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:57.879254 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.879162    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7727f857-7018-4cd0-ac20-6ea0917cf48c-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:36:57.879254 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:57.879194    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df5br\" (UniqueName: \"kubernetes.io/projected/7727f857-7018-4cd0-ac20-6ea0917cf48c-kube-api-access-df5br\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:36:58.048432 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:58.048399    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"]
Sep 15 21:36:58.052167 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:58.052138    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a276f9-77455d64c8-fp2hr"]
Sep 15 21:36:58.703077 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:58.703046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:36:59.905440 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:36:59.905402    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7727f857-7018-4cd0-ac20-6ea0917cf48c" path="/var/lib/kubelet/pods/7727f857-7018-4cd0-ac20-6ea0917cf48c/volumes"
Sep 15 21:37:01.687881 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:01.687852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:37:25.369299 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.369264    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"]
Sep 15 21:37:25.369908 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.369528    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz" podUID="d886e7db-d050-416c-b971-7eea8ff7173c" containerName="maas-api" containerID="cri-o://19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3" gracePeriod=30
Sep 15 21:37:25.435681 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.435648    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"]
Sep 15 21:37:25.435945 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.435912    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc" podUID="bfce212f-de75-488c-b573-cacfa5bb1553" containerName="payload-pre-processing" containerID="cri-o://fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b" gracePeriod=30
Sep 15 21:37:25.526294 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.526254    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc" podUID="bfce212f-de75-488c-b573-cacfa5bb1553" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.66:9004: connect: connection refused"
Sep 15 21:37:25.526294 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.526269    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz" podUID="d886e7db-d050-416c-b971-7eea8ff7173c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.65:8443/health\": dial tcp 10.132.0.65:8443: connect: connection refused"
Sep 15 21:37:25.705817 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.705791    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:37:25.733475 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.733142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:37:25.762818 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.762790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-metrics-certs\") pod \"d886e7db-d050-416c-b971-7eea8ff7173c\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") "
Sep 15 21:37:25.763000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.762837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwwbc\" (UniqueName: \"kubernetes.io/projected/d886e7db-d050-416c-b971-7eea8ff7173c-kube-api-access-hwwbc\") pod \"d886e7db-d050-416c-b971-7eea8ff7173c\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") "
Sep 15 21:37:25.763000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.762896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-maas-api-tls\") pod \"d886e7db-d050-416c-b971-7eea8ff7173c\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") "
Sep 15 21:37:25.763000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.762941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886e7db-d050-416c-b971-7eea8ff7173c-openshift-service-ca\") pod \"d886e7db-d050-416c-b971-7eea8ff7173c\" (UID: \"d886e7db-d050-416c-b971-7eea8ff7173c\") "
Sep 15 21:37:25.763632 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.763566    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d886e7db-d050-416c-b971-7eea8ff7173c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d886e7db-d050-416c-b971-7eea8ff7173c" (UID: "d886e7db-d050-416c-b971-7eea8ff7173c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:25.765827 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.765800    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d886e7db-d050-416c-b971-7eea8ff7173c" (UID: "d886e7db-d050-416c-b971-7eea8ff7173c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:37:25.765926 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.765913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d886e7db-d050-416c-b971-7eea8ff7173c" (UID: "d886e7db-d050-416c-b971-7eea8ff7173c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:37:25.765984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.765960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d886e7db-d050-416c-b971-7eea8ff7173c-kube-api-access-hwwbc" (OuterVolumeSpecName: "kube-api-access-hwwbc") pod "d886e7db-d050-416c-b971-7eea8ff7173c" (UID: "d886e7db-d050-416c-b971-7eea8ff7173c"). InnerVolumeSpecName "kube-api-access-hwwbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:25.864410 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.864380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffh46\" (UniqueName: \"kubernetes.io/projected/bfce212f-de75-488c-b573-cacfa5bb1553-kube-api-access-ffh46\") pod \"bfce212f-de75-488c-b573-cacfa5bb1553\" (UID: \"bfce212f-de75-488c-b573-cacfa5bb1553\") "
Sep 15 21:37:25.864410 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.864423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfce212f-de75-488c-b573-cacfa5bb1553-plugins-config-volume\") pod \"bfce212f-de75-488c-b573-cacfa5bb1553\" (UID: \"bfce212f-de75-488c-b573-cacfa5bb1553\") "
Sep 15 21:37:25.864684 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.864637    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwwbc\" (UniqueName: \"kubernetes.io/projected/d886e7db-d050-416c-b971-7eea8ff7173c-kube-api-access-hwwbc\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.864684 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.864650    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.864684 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.864658    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886e7db-d050-416c-b971-7eea8ff7173c-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.864684 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.864666    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d886e7db-d050-416c-b971-7eea8ff7173c-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.864916 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.864886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfce212f-de75-488c-b573-cacfa5bb1553-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfce212f-de75-488c-b573-cacfa5bb1553" (UID: "bfce212f-de75-488c-b573-cacfa5bb1553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:25.866592 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.866573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfce212f-de75-488c-b573-cacfa5bb1553-kube-api-access-ffh46" (OuterVolumeSpecName: "kube-api-access-ffh46") pod "bfce212f-de75-488c-b573-cacfa5bb1553" (UID: "bfce212f-de75-488c-b573-cacfa5bb1553"). InnerVolumeSpecName "kube-api-access-ffh46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:25.910504 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.910389    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfce212f-de75-488c-b573-cacfa5bb1553" containerID="fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b" exitCode=0
Sep 15 21:37:25.910670 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.910536    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"
Sep 15 21:37:25.912318 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.912290    2563 generic.go:356] "Generic (PLEG): container finished" podID="d886e7db-d050-416c-b971-7eea8ff7173c" containerID="19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3" exitCode=0
Sep 15 21:37:25.912490 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.912411    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"
Sep 15 21:37:25.914517 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.914487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc" event={"ID":"bfce212f-de75-488c-b573-cacfa5bb1553","Type":"ContainerDied","Data":"fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b"}
Sep 15 21:37:25.914649 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.914545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc" event={"ID":"bfce212f-de75-488c-b573-cacfa5bb1553","Type":"ContainerDied","Data":"f734866262006a40ef671217c88146ef3dace012dee150d18f9db52cbb3a8e2d"}
Sep 15 21:37:25.914649 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.914600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz" event={"ID":"d886e7db-d050-416c-b971-7eea8ff7173c","Type":"ContainerDied","Data":"19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3"}
Sep 15 21:37:25.914760 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.914630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz" event={"ID":"d886e7db-d050-416c-b971-7eea8ff7173c","Type":"ContainerDied","Data":"804625e6ea6278cd95eb7241037c22301892eaec06a8679438f8b710b7dc89fe"}
Sep 15 21:37:25.914760 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.914684    2563 scope.go:117] "RemoveContainer" containerID="fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b"
Sep 15 21:37:25.926653 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.926628    2563 scope.go:117] "RemoveContainer" containerID="fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b"
Sep 15 21:37:25.926952 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:37:25.926934    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b\": container with ID starting with fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b not found: ID does not exist" containerID="fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b"
Sep 15 21:37:25.927033 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.926957    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b"} err="failed to get container status \"fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b\": rpc error: code = NotFound desc = could not find container \"fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b\": container with ID starting with fff880200d23bc077515b2abb72fb3ae20c1e19785522bdcf0dc3eec167c818b not found: ID does not exist"
Sep 15 21:37:25.927033 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.926972    2563 scope.go:117] "RemoveContainer" containerID="19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3"
Sep 15 21:37:25.935364 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.935348    2563 scope.go:117] "RemoveContainer" containerID="19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3"
Sep 15 21:37:25.935661 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:37:25.935642    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3\": container with ID starting with 19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3 not found: ID does not exist" containerID="19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3"
Sep 15 21:37:25.935727 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.935668    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3"} err="failed to get container status \"19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3\": rpc error: code = NotFound desc = could not find container \"19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3\": container with ID starting with 19fd3ded6bdc478866366d9609e29185dced257d1c19695b31308834a004d1c3 not found: ID does not exist"
Sep 15 21:37:25.939707 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.939674    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"]
Sep 15 21:37:25.941816 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.941795    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0753ad-6fc647f6fd-l62hz"]
Sep 15 21:37:25.950388 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.950368    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"]
Sep 15 21:37:25.956867 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.956849    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0753ad-7478bfb454-8kknc"]
Sep 15 21:37:25.966306 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.966283    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffh46\" (UniqueName: \"kubernetes.io/projected/bfce212f-de75-488c-b573-cacfa5bb1553-kube-api-access-ffh46\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.966306 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:25.966304    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfce212f-de75-488c-b573-cacfa5bb1553-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:27.901612 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:27.901581    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfce212f-de75-488c-b573-cacfa5bb1553" path="/var/lib/kubelet/pods/bfce212f-de75-488c-b573-cacfa5bb1553/volumes"
Sep 15 21:37:27.901993 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:27.901973    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d886e7db-d050-416c-b971-7eea8ff7173c" path="/var/lib/kubelet/pods/d886e7db-d050-416c-b971-7eea8ff7173c/volumes"
Sep 15 21:37:48.705526 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:48.705492    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"]
Sep 15 21:37:48.706066 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:48.705684    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg" podUID="3f18b9b0-26b0-4459-8582-a75bb520cc48" containerName="maas-api" containerID="cri-o://eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4" gracePeriod=30
Sep 15 21:37:48.823129 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:48.823099    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"]
Sep 15 21:37:48.823419 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:48.823386    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj" podUID="3f69c8d4-3435-471b-8328-3aa1bfc4106a" containerName="payload-processing" containerID="cri-o://4499f3bdf6d07cac864b41e206c0f221be3d53a3875d924789426bb73c4b07cc" gracePeriod=30
Sep 15 21:37:48.990642 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:48.990613    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:37:49.003440 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.003408    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjrf9\" (UniqueName: \"kubernetes.io/projected/3f18b9b0-26b0-4459-8582-a75bb520cc48-kube-api-access-pjrf9\") pod \"3f18b9b0-26b0-4459-8582-a75bb520cc48\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") "
Sep 15 21:37:49.003633 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.003520    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls\") pod \"3f18b9b0-26b0-4459-8582-a75bb520cc48\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") "
Sep 15 21:37:49.003633 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.003593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f18b9b0-26b0-4459-8582-a75bb520cc48-openshift-service-ca\") pod \"3f18b9b0-26b0-4459-8582-a75bb520cc48\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") "
Sep 15 21:37:49.003633 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.003629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-metrics-certs\") pod \"3f18b9b0-26b0-4459-8582-a75bb520cc48\" (UID: \"3f18b9b0-26b0-4459-8582-a75bb520cc48\") "
Sep 15 21:37:49.004332 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.004122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f18b9b0-26b0-4459-8582-a75bb520cc48-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f18b9b0-26b0-4459-8582-a75bb520cc48" (UID: "3f18b9b0-26b0-4459-8582-a75bb520cc48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:49.006329 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.006269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f18b9b0-26b0-4459-8582-a75bb520cc48-kube-api-access-pjrf9" (OuterVolumeSpecName: "kube-api-access-pjrf9") pod "3f18b9b0-26b0-4459-8582-a75bb520cc48" (UID: "3f18b9b0-26b0-4459-8582-a75bb520cc48"). InnerVolumeSpecName "kube-api-access-pjrf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:49.008221 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.008098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f18b9b0-26b0-4459-8582-a75bb520cc48" (UID: "3f18b9b0-26b0-4459-8582-a75bb520cc48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:37:49.010732 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.010691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f18b9b0-26b0-4459-8582-a75bb520cc48" (UID: "3f18b9b0-26b0-4459-8582-a75bb520cc48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:37:49.048858 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.048821    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f18b9b0-26b0-4459-8582-a75bb520cc48" containerID="eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4" exitCode=0
Sep 15 21:37:49.049057 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.048874    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"
Sep 15 21:37:49.049057 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.048901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg" event={"ID":"3f18b9b0-26b0-4459-8582-a75bb520cc48","Type":"ContainerDied","Data":"eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4"}
Sep 15 21:37:49.049057 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.048963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg" event={"ID":"3f18b9b0-26b0-4459-8582-a75bb520cc48","Type":"ContainerDied","Data":"f0d18b32b96e10efc347c64cba9b45a434139910fc70bbf53c05fd6efe74fd29"}
Sep 15 21:37:49.049057 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.048994    2563 scope.go:117] "RemoveContainer" containerID="eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4"
Sep 15 21:37:49.051636 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.051602    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f69c8d4-3435-471b-8328-3aa1bfc4106a" containerID="4499f3bdf6d07cac864b41e206c0f221be3d53a3875d924789426bb73c4b07cc" exitCode=0
Sep 15 21:37:49.051799 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.051668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj" event={"ID":"3f69c8d4-3435-471b-8328-3aa1bfc4106a","Type":"ContainerDied","Data":"4499f3bdf6d07cac864b41e206c0f221be3d53a3875d924789426bb73c4b07cc"}
Sep 15 21:37:49.062708 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.062586    2563 scope.go:117] "RemoveContainer" containerID="eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4"
Sep 15 21:37:49.063174 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:37:49.062999    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4\": container with ID starting with eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4 not found: ID does not exist" containerID="eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4"
Sep 15 21:37:49.063174 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.063055    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4"} err="failed to get container status \"eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4\": rpc error: code = NotFound desc = could not find container \"eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4\": container with ID starting with eea9e6d6ec4fba5531aabe3a0f4a0f7853a3c7cd3d334ec8259e5c47a1c3bcb4 not found: ID does not exist"
Sep 15 21:37:49.074914 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.074863    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"]
Sep 15 21:37:49.079123 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.079097    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30e32e-776f9df9d9-4g2rg"]
Sep 15 21:37:49.098668 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.098640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:37:49.105300 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.105274    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjrf9\" (UniqueName: \"kubernetes.io/projected/3f18b9b0-26b0-4459-8582-a75bb520cc48-kube-api-access-pjrf9\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.105300 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.105299    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.105542 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.105309    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f18b9b0-26b0-4459-8582-a75bb520cc48-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.105542 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.105317    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f18b9b0-26b0-4459-8582-a75bb520cc48-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.206172 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.206134    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbkdk\" (UniqueName: \"kubernetes.io/projected/3f69c8d4-3435-471b-8328-3aa1bfc4106a-kube-api-access-fbkdk\") pod \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\" (UID: \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\") "
Sep 15 21:37:49.206392 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.206201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69c8d4-3435-471b-8328-3aa1bfc4106a-plugins-config-volume\") pod \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\" (UID: \"3f69c8d4-3435-471b-8328-3aa1bfc4106a\") "
Sep 15 21:37:49.206627 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.206599    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f69c8d4-3435-471b-8328-3aa1bfc4106a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f69c8d4-3435-471b-8328-3aa1bfc4106a" (UID: "3f69c8d4-3435-471b-8328-3aa1bfc4106a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:49.208710 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.208655    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f69c8d4-3435-471b-8328-3aa1bfc4106a-kube-api-access-fbkdk" (OuterVolumeSpecName: "kube-api-access-fbkdk") pod "3f69c8d4-3435-471b-8328-3aa1bfc4106a" (UID: "3f69c8d4-3435-471b-8328-3aa1bfc4106a"). InnerVolumeSpecName "kube-api-access-fbkdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:49.307721 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.307684    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbkdk\" (UniqueName: \"kubernetes.io/projected/3f69c8d4-3435-471b-8328-3aa1bfc4106a-kube-api-access-fbkdk\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.307721 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.307717    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69c8d4-3435-471b-8328-3aa1bfc4106a-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.902017 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:49.901985    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f18b9b0-26b0-4459-8582-a75bb520cc48" path="/var/lib/kubelet/pods/3f18b9b0-26b0-4459-8582-a75bb520cc48/volumes"
Sep 15 21:37:50.060505 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:50.060389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj" event={"ID":"3f69c8d4-3435-471b-8328-3aa1bfc4106a","Type":"ContainerDied","Data":"b155bc23de72bc025fdfa0bf5900d55e7c8bb8c92be7ec30e52628f265c39030"}
Sep 15 21:37:50.060505 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:50.060425    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"
Sep 15 21:37:50.060505 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:50.060479    2563 scope.go:117] "RemoveContainer" containerID="4499f3bdf6d07cac864b41e206c0f221be3d53a3875d924789426bb73c4b07cc"
Sep 15 21:37:50.086588 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:50.086559    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"]
Sep 15 21:37:50.091457 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:50.091404    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30e32e-6d597fbbc7-fktlj"]
Sep 15 21:37:51.904267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:37:51.904234    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f69c8d4-3435-471b-8328-3aa1bfc4106a" path="/var/lib/kubelet/pods/3f69c8d4-3435-471b-8328-3aa1bfc4106a/volumes"
Sep 15 21:38:08.874397 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:08.874360    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"]
Sep 15 21:38:08.875205 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:08.875167    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz" podUID="8b4b182b-50a4-4af9-a218-49a775acde60" containerName="maas-api" containerID="cri-o://52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e" gracePeriod=30
Sep 15 21:38:09.164693 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.164666    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:38:09.175771 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.175729    2563 generic.go:356] "Generic (PLEG): container finished" podID="8b4b182b-50a4-4af9-a218-49a775acde60" containerID="52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e" exitCode=0
Sep 15 21:38:09.175941 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.175772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz" event={"ID":"8b4b182b-50a4-4af9-a218-49a775acde60","Type":"ContainerDied","Data":"52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e"}
Sep 15 21:38:09.175941 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.175800    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"
Sep 15 21:38:09.175941 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.175838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz" event={"ID":"8b4b182b-50a4-4af9-a218-49a775acde60","Type":"ContainerDied","Data":"af1c745ef0e27c2079fddf380390833dfea927c404ecec7530c7671d296fa579"}
Sep 15 21:38:09.175941 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.175868    2563 scope.go:117] "RemoveContainer" containerID="52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e"
Sep 15 21:38:09.192765 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.192735    2563 scope.go:117] "RemoveContainer" containerID="52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e"
Sep 15 21:38:09.193208 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:38:09.193178    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e\": container with ID starting with 52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e not found: ID does not exist" containerID="52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e"
Sep 15 21:38:09.193337 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.193217    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e"} err="failed to get container status \"52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e\": rpc error: code = NotFound desc = could not find container \"52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e\": container with ID starting with 52c9a8d5e91051eef1aaf6e5fac79df58161e223aef325e36b592c9244b8fb1e not found: ID does not exist"
Sep 15 21:38:09.322791 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.322754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-maas-api-tls\") pod \"8b4b182b-50a4-4af9-a218-49a775acde60\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") "
Sep 15 21:38:09.323010 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.322809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76jzd\" (UniqueName: \"kubernetes.io/projected/8b4b182b-50a4-4af9-a218-49a775acde60-kube-api-access-76jzd\") pod \"8b4b182b-50a4-4af9-a218-49a775acde60\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") "
Sep 15 21:38:09.323010 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.322954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b182b-50a4-4af9-a218-49a775acde60-openshift-service-ca\") pod \"8b4b182b-50a4-4af9-a218-49a775acde60\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") "
Sep 15 21:38:09.323010 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.323001    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-metrics-certs\") pod \"8b4b182b-50a4-4af9-a218-49a775acde60\" (UID: \"8b4b182b-50a4-4af9-a218-49a775acde60\") "
Sep 15 21:38:09.323298 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.323254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4b182b-50a4-4af9-a218-49a775acde60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b4b182b-50a4-4af9-a218-49a775acde60" (UID: "8b4b182b-50a4-4af9-a218-49a775acde60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:09.323390 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.323334    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b182b-50a4-4af9-a218-49a775acde60-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.325966 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.325727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4b182b-50a4-4af9-a218-49a775acde60-kube-api-access-76jzd" (OuterVolumeSpecName: "kube-api-access-76jzd") pod "8b4b182b-50a4-4af9-a218-49a775acde60" (UID: "8b4b182b-50a4-4af9-a218-49a775acde60"). InnerVolumeSpecName "kube-api-access-76jzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:09.325966 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.325923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8b4b182b-50a4-4af9-a218-49a775acde60" (UID: "8b4b182b-50a4-4af9-a218-49a775acde60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:09.326432 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.326405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b4b182b-50a4-4af9-a218-49a775acde60" (UID: "8b4b182b-50a4-4af9-a218-49a775acde60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:09.424177 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.424085    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.424177 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.424118    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b4b182b-50a4-4af9-a218-49a775acde60-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.424177 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.424128    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76jzd\" (UniqueName: \"kubernetes.io/projected/8b4b182b-50a4-4af9-a218-49a775acde60-kube-api-access-76jzd\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.497697 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.497665    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"]
Sep 15 21:38:09.502412 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.502385    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a3aa55-5557b97ff9-6ttsz"]
Sep 15 21:38:09.902715 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:09.902675    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b4b182b-50a4-4af9-a218-49a775acde60" path="/var/lib/kubelet/pods/8b4b182b-50a4-4af9-a218-49a775acde60/volumes"
Sep 15 21:38:18.812524 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:18.812485    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"]
Sep 15 21:38:18.813029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:18.812715    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m" podUID="a4c753f4-7f52-4867-8345-6a0efef1be86" containerName="maas-api" containerID="cri-o://d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514" gracePeriod=30
Sep 15 21:38:18.871228 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:18.871191    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"]
Sep 15 21:38:18.871487 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:18.871440    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z" podUID="a176feb7-0805-4413-a305-07fe3130200a" containerName="payload-processing" containerID="cri-o://9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62" gracePeriod=30
Sep 15 21:38:19.165705 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.165679    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:38:19.200518 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.200479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:38:19.228469 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.228423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c753f4-7f52-4867-8345-6a0efef1be86-openshift-service-ca\") pod \"a4c753f4-7f52-4867-8345-6a0efef1be86\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") "
Sep 15 21:38:19.228645 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.228511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pm2c\" (UniqueName: \"kubernetes.io/projected/a176feb7-0805-4413-a305-07fe3130200a-kube-api-access-6pm2c\") pod \"a176feb7-0805-4413-a305-07fe3130200a\" (UID: \"a176feb7-0805-4413-a305-07fe3130200a\") "
Sep 15 21:38:19.228645 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.228538    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-metrics-certs\") pod \"a4c753f4-7f52-4867-8345-6a0efef1be86\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") "
Sep 15 21:38:19.228645 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.228571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a176feb7-0805-4413-a305-07fe3130200a-plugins-config-volume\") pod \"a176feb7-0805-4413-a305-07fe3130200a\" (UID: \"a176feb7-0805-4413-a305-07fe3130200a\") "
Sep 15 21:38:19.228824 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.228667    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vbnq\" (UniqueName: \"kubernetes.io/projected/a4c753f4-7f52-4867-8345-6a0efef1be86-kube-api-access-4vbnq\") pod \"a4c753f4-7f52-4867-8345-6a0efef1be86\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") "
Sep 15 21:38:19.228824 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.228733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls\") pod \"a4c753f4-7f52-4867-8345-6a0efef1be86\" (UID: \"a4c753f4-7f52-4867-8345-6a0efef1be86\") "
Sep 15 21:38:19.228824 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.228768    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c753f4-7f52-4867-8345-6a0efef1be86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4c753f4-7f52-4867-8345-6a0efef1be86" (UID: "a4c753f4-7f52-4867-8345-6a0efef1be86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:19.229037 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.229013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a176feb7-0805-4413-a305-07fe3130200a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a176feb7-0805-4413-a305-07fe3130200a" (UID: "a176feb7-0805-4413-a305-07fe3130200a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:19.229114 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.229092    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c753f4-7f52-4867-8345-6a0efef1be86-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:19.231431 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.231393    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4c753f4-7f52-4867-8345-6a0efef1be86" (UID: "a4c753f4-7f52-4867-8345-6a0efef1be86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:19.232380 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.231797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a176feb7-0805-4413-a305-07fe3130200a-kube-api-access-6pm2c" (OuterVolumeSpecName: "kube-api-access-6pm2c") pod "a176feb7-0805-4413-a305-07fe3130200a" (UID: "a176feb7-0805-4413-a305-07fe3130200a"). InnerVolumeSpecName "kube-api-access-6pm2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:19.232380 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.232037    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c753f4-7f52-4867-8345-6a0efef1be86-kube-api-access-4vbnq" (OuterVolumeSpecName: "kube-api-access-4vbnq") pod "a4c753f4-7f52-4867-8345-6a0efef1be86" (UID: "a4c753f4-7f52-4867-8345-6a0efef1be86"). InnerVolumeSpecName "kube-api-access-4vbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:19.232543 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.232405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4c753f4-7f52-4867-8345-6a0efef1be86" (UID: "a4c753f4-7f52-4867-8345-6a0efef1be86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:19.252480 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.252415    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4c753f4-7f52-4867-8345-6a0efef1be86" containerID="d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514" exitCode=0
Sep 15 21:38:19.252679 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.252478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m" event={"ID":"a4c753f4-7f52-4867-8345-6a0efef1be86","Type":"ContainerDied","Data":"d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514"}
Sep 15 21:38:19.252679 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.252501    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"
Sep 15 21:38:19.252679 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.252566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m" event={"ID":"a4c753f4-7f52-4867-8345-6a0efef1be86","Type":"ContainerDied","Data":"9552c435489f7ff664e265e0dfa3ef5925e6e5603925e49e1fcd259f6e338650"}
Sep 15 21:38:19.252679 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.252596    2563 scope.go:117] "RemoveContainer" containerID="d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514"
Sep 15 21:38:19.254087 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.254062    2563 generic.go:356] "Generic (PLEG): container finished" podID="a176feb7-0805-4413-a305-07fe3130200a" containerID="9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62" exitCode=0
Sep 15 21:38:19.254203 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.254110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z" event={"ID":"a176feb7-0805-4413-a305-07fe3130200a","Type":"ContainerDied","Data":"9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62"}
Sep 15 21:38:19.254203 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.254150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z" event={"ID":"a176feb7-0805-4413-a305-07fe3130200a","Type":"ContainerDied","Data":"1bdc90de0c8ea0b4e96562e185c49ac20c6a6c7fafd9a0ff8350996d42ab9f19"}
Sep 15 21:38:19.254330 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.254232    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"
Sep 15 21:38:19.263228 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.263203    2563 scope.go:117] "RemoveContainer" containerID="d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514"
Sep 15 21:38:19.263716 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:38:19.263688    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514\": container with ID starting with d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514 not found: ID does not exist" containerID="d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514"
Sep 15 21:38:19.263885 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.263730    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514"} err="failed to get container status \"d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514\": rpc error: code = NotFound desc = could not find container \"d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514\": container with ID starting with d8d368bf36c32456a1758912c6be50ad095913a2cdfa33d2d2ee37a02e88c514 not found: ID does not exist"
Sep 15 21:38:19.263885 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.263755    2563 scope.go:117] "RemoveContainer" containerID="9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62"
Sep 15 21:38:19.272554 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.272534    2563 scope.go:117] "RemoveContainer" containerID="9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62"
Sep 15 21:38:19.272839 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:38:19.272819    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62\": container with ID starting with 9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62 not found: ID does not exist" containerID="9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62"
Sep 15 21:38:19.272891 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.272847    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62"} err="failed to get container status \"9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62\": rpc error: code = NotFound desc = could not find container \"9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62\": container with ID starting with 9a5fedceb65d7165bf2cef5c0f12ceb8cdefc5666c6c7a0166b7c79801bb3d62 not found: ID does not exist"
Sep 15 21:38:19.279929 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.279905    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"]
Sep 15 21:38:19.284259 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.284238    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-625609-956b54778-zz69m"]
Sep 15 21:38:19.297237 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.297209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"]
Sep 15 21:38:19.304705 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.304676    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-625609-5d8d694859-vwr5z"]
Sep 15 21:38:19.330470 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.330412    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pm2c\" (UniqueName: \"kubernetes.io/projected/a176feb7-0805-4413-a305-07fe3130200a-kube-api-access-6pm2c\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:19.330470 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.330459    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:19.330470 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.330471    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a176feb7-0805-4413-a305-07fe3130200a-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:19.330470 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.330479    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vbnq\" (UniqueName: \"kubernetes.io/projected/a4c753f4-7f52-4867-8345-6a0efef1be86-kube-api-access-4vbnq\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:19.330761 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.330487    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c753f4-7f52-4867-8345-6a0efef1be86-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:19.902285 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.902252    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a176feb7-0805-4413-a305-07fe3130200a" path="/var/lib/kubelet/pods/a176feb7-0805-4413-a305-07fe3130200a/volumes"
Sep 15 21:38:19.902714 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:19.902683    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c753f4-7f52-4867-8345-6a0efef1be86" path="/var/lib/kubelet/pods/a4c753f4-7f52-4867-8345-6a0efef1be86/volumes"
Sep 15 21:38:27.191418 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.191386    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"]
Sep 15 21:38:27.191912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.191644    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz" podUID="e4424a83-cdaf-4cd0-a403-55621bc9938d" containerName="maas-api" containerID="cri-o://c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789" gracePeriod=30
Sep 15 21:38:27.446870 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.446801    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:38:27.509293 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.509257    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls\") pod \"e4424a83-cdaf-4cd0-a403-55621bc9938d\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") "
Sep 15 21:38:27.509293 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.509295    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4424a83-cdaf-4cd0-a403-55621bc9938d-openshift-service-ca\") pod \"e4424a83-cdaf-4cd0-a403-55621bc9938d\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") "
Sep 15 21:38:27.509581 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.509333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-metrics-certs\") pod \"e4424a83-cdaf-4cd0-a403-55621bc9938d\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") "
Sep 15 21:38:27.509581 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.509499    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q54m\" (UniqueName: \"kubernetes.io/projected/e4424a83-cdaf-4cd0-a403-55621bc9938d-kube-api-access-5q54m\") pod \"e4424a83-cdaf-4cd0-a403-55621bc9938d\" (UID: \"e4424a83-cdaf-4cd0-a403-55621bc9938d\") "
Sep 15 21:38:27.509726 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.509698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4424a83-cdaf-4cd0-a403-55621bc9938d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4424a83-cdaf-4cd0-a403-55621bc9938d" (UID: "e4424a83-cdaf-4cd0-a403-55621bc9938d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:27.509923 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.509908    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4424a83-cdaf-4cd0-a403-55621bc9938d-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:27.511731 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.511706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4424a83-cdaf-4cd0-a403-55621bc9938d" (UID: "e4424a83-cdaf-4cd0-a403-55621bc9938d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:27.511867 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.511728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4424a83-cdaf-4cd0-a403-55621bc9938d-kube-api-access-5q54m" (OuterVolumeSpecName: "kube-api-access-5q54m") pod "e4424a83-cdaf-4cd0-a403-55621bc9938d" (UID: "e4424a83-cdaf-4cd0-a403-55621bc9938d"). InnerVolumeSpecName "kube-api-access-5q54m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:27.511867 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.511817    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e4424a83-cdaf-4cd0-a403-55621bc9938d" (UID: "e4424a83-cdaf-4cd0-a403-55621bc9938d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:27.611297 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.611263    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:27.611297 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.611294    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q54m\" (UniqueName: \"kubernetes.io/projected/e4424a83-cdaf-4cd0-a403-55621bc9938d-kube-api-access-5q54m\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:27.611297 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:27.611304    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4424a83-cdaf-4cd0-a403-55621bc9938d-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:38:28.309764 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.309729    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4424a83-cdaf-4cd0-a403-55621bc9938d" containerID="c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789" exitCode=0
Sep 15 21:38:28.310290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.309770    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"
Sep 15 21:38:28.310290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.309812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz" event={"ID":"e4424a83-cdaf-4cd0-a403-55621bc9938d","Type":"ContainerDied","Data":"c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789"}
Sep 15 21:38:28.310290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.309869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz" event={"ID":"e4424a83-cdaf-4cd0-a403-55621bc9938d","Type":"ContainerDied","Data":"a74524410e6567116556499a95930c135e8e96cbfbcfe91133687090bf9cb806"}
Sep 15 21:38:28.310290 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.309894    2563 scope.go:117] "RemoveContainer" containerID="c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789"
Sep 15 21:38:28.320322 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.320302    2563 scope.go:117] "RemoveContainer" containerID="c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789"
Sep 15 21:38:28.320712 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:38:28.320688    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789\": container with ID starting with c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789 not found: ID does not exist" containerID="c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789"
Sep 15 21:38:28.320787 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.320723    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789"} err="failed to get container status \"c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789\": rpc error: code = NotFound desc = could not find container \"c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789\": container with ID starting with c0b1f916c0fab6ca72ddf83d927c6636307ae90d1f4c6acd72f1013702bed789 not found: ID does not exist"
Sep 15 21:38:28.325752 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.325728    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"]
Sep 15 21:38:28.329333 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:28.329307    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9aeead-5bf867ddf5-4tbtz"]
Sep 15 21:38:29.903237 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:38:29.903201    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4424a83-cdaf-4cd0-a403-55621bc9938d" path="/var/lib/kubelet/pods/e4424a83-cdaf-4cd0-a403-55621bc9938d/volumes"
Sep 15 21:42:40.619089 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:42:40.619060    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fsqhb_023c1a26-91e1-4d9a-8871-fe737cdac0c5/payload-processing/0.log"
Sep 15 21:43:21.733779 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:21.733741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fsqhb_023c1a26-91e1-4d9a-8871-fe737cdac0c5/payload-processing/0.log"
Sep 15 21:43:28.570954 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.570919    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"]
Sep 15 21:43:28.571403 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571389    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4424a83-cdaf-4cd0-a403-55621bc9938d" containerName="maas-api"
Sep 15 21:43:28.571466 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571408    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4424a83-cdaf-4cd0-a403-55621bc9938d" containerName="maas-api"
Sep 15 21:43:28.571466 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571422    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c753f4-7f52-4867-8345-6a0efef1be86" containerName="maas-api"
Sep 15 21:43:28.571466 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571428    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c753f4-7f52-4867-8345-6a0efef1be86" containerName="maas-api"
Sep 15 21:43:28.571466 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a176feb7-0805-4413-a305-07fe3130200a" containerName="payload-processing"
Sep 15 21:43:28.571466 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571458    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a176feb7-0805-4413-a305-07fe3130200a" containerName="payload-processing"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4186d9f9-39af-418d-8c62-26d8a81bb6fa" containerName="main"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571475    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4186d9f9-39af-418d-8c62-26d8a81bb6fa" containerName="main"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571487    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d886e7db-d050-416c-b971-7eea8ff7173c" containerName="maas-api"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571495    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d886e7db-d050-416c-b971-7eea8ff7173c" containerName="maas-api"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571501    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfce212f-de75-488c-b573-cacfa5bb1553" containerName="payload-pre-processing"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571506    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfce212f-de75-488c-b573-cacfa5bb1553" containerName="payload-pre-processing"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571518    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b4b182b-50a4-4af9-a218-49a775acde60" containerName="maas-api"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571522    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b4b182b-50a4-4af9-a218-49a775acde60" containerName="maas-api"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571530    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7727f857-7018-4cd0-ac20-6ea0917cf48c" containerName="payload-pre-processing"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7727f857-7018-4cd0-ac20-6ea0917cf48c" containerName="payload-pre-processing"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571547    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f69c8d4-3435-471b-8328-3aa1bfc4106a" containerName="payload-processing"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f69c8d4-3435-471b-8328-3aa1bfc4106a" containerName="payload-processing"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571579    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f18b9b0-26b0-4459-8582-a75bb520cc48" containerName="maas-api"
Sep 15 21:43:28.571659 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571590    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f18b9b0-26b0-4459-8582-a75bb520cc48" containerName="maas-api"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571707    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b4b182b-50a4-4af9-a218-49a775acde60" containerName="maas-api"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571721    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f18b9b0-26b0-4459-8582-a75bb520cc48" containerName="maas-api"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571731    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7727f857-7018-4cd0-ac20-6ea0917cf48c" containerName="payload-pre-processing"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571739    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a176feb7-0805-4413-a305-07fe3130200a" containerName="payload-processing"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571745    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4424a83-cdaf-4cd0-a403-55621bc9938d" containerName="maas-api"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571751    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d886e7db-d050-416c-b971-7eea8ff7173c" containerName="maas-api"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571757    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c753f4-7f52-4867-8345-6a0efef1be86" containerName="maas-api"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571762    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f69c8d4-3435-471b-8328-3aa1bfc4106a" containerName="payload-processing"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571769    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4186d9f9-39af-418d-8c62-26d8a81bb6fa" containerName="main"
Sep 15 21:43:28.572144 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.571774    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfce212f-de75-488c-b573-cacfa5bb1553" containerName="payload-pre-processing"
Sep 15 21:43:28.575071 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.575054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.577814 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.577785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-494d21-serving-cert\""
Sep 15 21:43:28.585497 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.585472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"]
Sep 15 21:43:28.610528 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.610492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.610648 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.610608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.610648 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.610632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbwb\" (UniqueName: \"kubernetes.io/projected/72213fc9-f840-4bfb-b6b5-0f0258b185a2-kube-api-access-dsbwb\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.610730 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.610695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72213fc9-f840-4bfb-b6b5-0f0258b185a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.711941 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.711903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.712116 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.711973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.712116 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.712000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbwb\" (UniqueName: \"kubernetes.io/projected/72213fc9-f840-4bfb-b6b5-0f0258b185a2-kube-api-access-dsbwb\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.712214 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.712118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72213fc9-f840-4bfb-b6b5-0f0258b185a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.712745 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.712722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72213fc9-f840-4bfb-b6b5-0f0258b185a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.714537 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.714519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.714630 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.714614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.720399 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.720380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbwb\" (UniqueName: \"kubernetes.io/projected/72213fc9-f840-4bfb-b6b5-0f0258b185a2-kube-api-access-dsbwb\") pod \"maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:28.888060 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:28.887976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:29.171924 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.171898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"]
Sep 15 21:43:29.174260 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:43:29.174225    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72213fc9_f840_4bfb_b6b5_0f0258b185a2.slice/crio-09bb6efb08586b4aa259861b6e787924b2462fe70e69f159f333d7e385a4fe7c WatchSource:0}: Error finding container 09bb6efb08586b4aa259861b6e787924b2462fe70e69f159f333d7e385a4fe7c: Status 404 returned error can't find the container with id 09bb6efb08586b4aa259861b6e787924b2462fe70e69f159f333d7e385a4fe7c
Sep 15 21:43:29.175719 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.175701    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:43:29.263529 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.263494    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"]
Sep 15 21:43:29.268538 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.268519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:29.271113 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.271091    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-494d21\""
Sep 15 21:43:29.271286 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.271266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-494d21-dockercfg-rqlzn\""
Sep 15 21:43:29.279526 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.279503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"]
Sep 15 21:43:29.317410 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.317382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smf7n\" (UniqueName: \"kubernetes.io/projected/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-kube-api-access-smf7n\") pod \"payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb\" (UID: \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:29.317619 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.317423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb\" (UID: \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:29.418909 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.418871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smf7n\" (UniqueName: \"kubernetes.io/projected/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-kube-api-access-smf7n\") pod \"payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb\" (UID: \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:29.418909 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.418910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb\" (UID: \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:29.419713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.419696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb\" (UID: \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:29.427428 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.427366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smf7n\" (UniqueName: \"kubernetes.io/projected/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-kube-api-access-smf7n\") pod \"payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb\" (UID: \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:29.581664 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.581643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:29.861233 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.861205    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"]
Sep 15 21:43:29.863419 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:43:29.863390    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8712887_51a5_4092_9ab6_a4ef8bbac6a5.slice/crio-9ac4feff27eec03b9eb3126ed9ecf8affbbbe116a79a2b94c5691052d94e01d9 WatchSource:0}: Error finding container 9ac4feff27eec03b9eb3126ed9ecf8affbbbe116a79a2b94c5691052d94e01d9: Status 404 returned error can't find the container with id 9ac4feff27eec03b9eb3126ed9ecf8affbbbe116a79a2b94c5691052d94e01d9
Sep 15 21:43:29.933503 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.933433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb" event={"ID":"b8712887-51a5-4092-9ab6-a4ef8bbac6a5","Type":"ContainerStarted","Data":"9ac4feff27eec03b9eb3126ed9ecf8affbbbe116a79a2b94c5691052d94e01d9"}
Sep 15 21:43:29.935395 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.935369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82" event={"ID":"72213fc9-f840-4bfb-b6b5-0f0258b185a2","Type":"ContainerStarted","Data":"d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506"}
Sep 15 21:43:29.935722 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.935421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82" event={"ID":"72213fc9-f840-4bfb-b6b5-0f0258b185a2","Type":"ContainerStarted","Data":"09bb6efb08586b4aa259861b6e787924b2462fe70e69f159f333d7e385a4fe7c"}
Sep 15 21:43:29.974940 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:29.974893    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82" podStartSLOduration=1.57563313 podStartE2EDuration="1.974871875s" totalImagesPullingTime="399.238745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:29.971677907 +0000 UTC m=+2128.571590016" watchObservedRunningTime="2026-09-15 21:43:29.974871875 +0000 UTC m=+2128.574783988"
Sep 15 21:43:30.942282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:30.942227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb" event={"ID":"b8712887-51a5-4092-9ab6-a4ef8bbac6a5","Type":"ContainerStarted","Data":"860550037c660af9270d6987668910b091c1cbe318cbd05c7b6b595a98ff7539"}
Sep 15 21:43:30.942713 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:30.942364    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:30.959835 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:30.959781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb" podStartSLOduration=1.959765634 podStartE2EDuration="1.959765634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:30.958372924 +0000 UTC m=+2129.558285034" watchObservedRunningTime="2026-09-15 21:43:30.959765634 +0000 UTC m=+2129.559677743"
Sep 15 21:43:31.947709 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:31.947675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:43:36.952858 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:36.952827    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:43:42.955357 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:43:42.955313    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:44:06.818853 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:06.818754    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"]
Sep 15 21:44:06.819463 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:06.819038    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82" podUID="72213fc9-f840-4bfb-b6b5-0f0258b185a2" containerName="maas-api" containerID="cri-o://d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506" gracePeriod=30
Sep 15 21:44:06.878810 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:06.878772    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"]
Sep 15 21:44:06.879049 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:06.879022    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb" podUID="b8712887-51a5-4092-9ab6-a4ef8bbac6a5" containerName="payload-processing" containerID="cri-o://860550037c660af9270d6987668910b091c1cbe318cbd05c7b6b595a98ff7539" gracePeriod=30
Sep 15 21:44:06.948345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:06.948304    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82" podUID="72213fc9-f840-4bfb-b6b5-0f0258b185a2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.69:8443/health\": dial tcp 10.132.0.69:8443: connect: connection refused"
Sep 15 21:44:07.141820 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.141783    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:44:07.146181 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.146157    2563 generic.go:356] "Generic (PLEG): container finished" podID="72213fc9-f840-4bfb-b6b5-0f0258b185a2" containerID="d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506" exitCode=0
Sep 15 21:44:07.146318 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.146220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"
Sep 15 21:44:07.146318 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.146242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82" event={"ID":"72213fc9-f840-4bfb-b6b5-0f0258b185a2","Type":"ContainerDied","Data":"d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506"}
Sep 15 21:44:07.146318 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.146303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82" event={"ID":"72213fc9-f840-4bfb-b6b5-0f0258b185a2","Type":"ContainerDied","Data":"09bb6efb08586b4aa259861b6e787924b2462fe70e69f159f333d7e385a4fe7c"}
Sep 15 21:44:07.146513 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.146324    2563 scope.go:117] "RemoveContainer" containerID="d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506"
Sep 15 21:44:07.148258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.148234    2563 generic.go:356] "Generic (PLEG): container finished" podID="b8712887-51a5-4092-9ab6-a4ef8bbac6a5" containerID="860550037c660af9270d6987668910b091c1cbe318cbd05c7b6b595a98ff7539" exitCode=0
Sep 15 21:44:07.148367 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.148263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb" event={"ID":"b8712887-51a5-4092-9ab6-a4ef8bbac6a5","Type":"ContainerDied","Data":"860550037c660af9270d6987668910b091c1cbe318cbd05c7b6b595a98ff7539"}
Sep 15 21:44:07.156669 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.156529    2563 scope.go:117] "RemoveContainer" containerID="d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506"
Sep 15 21:44:07.156868 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:44:07.156839    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506\": container with ID starting with d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506 not found: ID does not exist" containerID="d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506"
Sep 15 21:44:07.156953 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.156875    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506"} err="failed to get container status \"d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506\": rpc error: code = NotFound desc = could not find container \"d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506\": container with ID starting with d20bf342322336443e3883c8d3334e641b71e81be78280cf7a7fa547c26a7506 not found: ID does not exist"
Sep 15 21:44:07.169283 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.168666    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-maas-api-tls\") pod \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") "
Sep 15 21:44:07.169283 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.168705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72213fc9-f840-4bfb-b6b5-0f0258b185a2-openshift-service-ca\") pod \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") "
Sep 15 21:44:07.169283 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.169179    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72213fc9-f840-4bfb-b6b5-0f0258b185a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72213fc9-f840-4bfb-b6b5-0f0258b185a2" (UID: "72213fc9-f840-4bfb-b6b5-0f0258b185a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:07.169283 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.169240    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-metrics-certs\") pod \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") "
Sep 15 21:44:07.169984 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.169954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsbwb\" (UniqueName: \"kubernetes.io/projected/72213fc9-f840-4bfb-b6b5-0f0258b185a2-kube-api-access-dsbwb\") pod \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\" (UID: \"72213fc9-f840-4bfb-b6b5-0f0258b185a2\") "
Sep 15 21:44:07.170393 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.170369    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72213fc9-f840-4bfb-b6b5-0f0258b185a2-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:44:07.171803 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.171779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72213fc9-f840-4bfb-b6b5-0f0258b185a2" (UID: "72213fc9-f840-4bfb-b6b5-0f0258b185a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:07.172380 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.172350    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72213fc9-f840-4bfb-b6b5-0f0258b185a2-kube-api-access-dsbwb" (OuterVolumeSpecName: "kube-api-access-dsbwb") pod "72213fc9-f840-4bfb-b6b5-0f0258b185a2" (UID: "72213fc9-f840-4bfb-b6b5-0f0258b185a2"). InnerVolumeSpecName "kube-api-access-dsbwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:07.172522 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.172399    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "72213fc9-f840-4bfb-b6b5-0f0258b185a2" (UID: "72213fc9-f840-4bfb-b6b5-0f0258b185a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:07.180851 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.180818    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:44:07.271164 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.271131    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-plugins-config-volume\") pod \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\" (UID: \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\") "
Sep 15 21:44:07.271354 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.271203    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smf7n\" (UniqueName: \"kubernetes.io/projected/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-kube-api-access-smf7n\") pod \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\" (UID: \"b8712887-51a5-4092-9ab6-a4ef8bbac6a5\") "
Sep 15 21:44:07.271434 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.271356    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:44:07.271434 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.271366    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72213fc9-f840-4bfb-b6b5-0f0258b185a2-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:44:07.271434 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.271374    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsbwb\" (UniqueName: \"kubernetes.io/projected/72213fc9-f840-4bfb-b6b5-0f0258b185a2-kube-api-access-dsbwb\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:44:07.271634 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.271526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8712887-51a5-4092-9ab6-a4ef8bbac6a5" (UID: "b8712887-51a5-4092-9ab6-a4ef8bbac6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:07.274037 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.274010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-kube-api-access-smf7n" (OuterVolumeSpecName: "kube-api-access-smf7n") pod "b8712887-51a5-4092-9ab6-a4ef8bbac6a5" (UID: "b8712887-51a5-4092-9ab6-a4ef8bbac6a5"). InnerVolumeSpecName "kube-api-access-smf7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:07.372910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.372829    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:44:07.372910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.372859    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smf7n\" (UniqueName: \"kubernetes.io/projected/b8712887-51a5-4092-9ab6-a4ef8bbac6a5-kube-api-access-smf7n\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:44:07.467854 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.467821    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"]
Sep 15 21:44:07.471828 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.471803    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-494d21-7ffc5c6f96-26m82"]
Sep 15 21:44:07.903543 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:07.903514    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72213fc9-f840-4bfb-b6b5-0f0258b185a2" path="/var/lib/kubelet/pods/72213fc9-f840-4bfb-b6b5-0f0258b185a2/volumes"
Sep 15 21:44:08.156091 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:08.156000    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"
Sep 15 21:44:08.156091 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:08.156015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb" event={"ID":"b8712887-51a5-4092-9ab6-a4ef8bbac6a5","Type":"ContainerDied","Data":"9ac4feff27eec03b9eb3126ed9ecf8affbbbe116a79a2b94c5691052d94e01d9"}
Sep 15 21:44:08.156091 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:08.156089    2563 scope.go:117] "RemoveContainer" containerID="860550037c660af9270d6987668910b091c1cbe318cbd05c7b6b595a98ff7539"
Sep 15 21:44:08.182440 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:08.182396    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"]
Sep 15 21:44:08.187425 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:08.187396    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-494d21-fc579d7cf-j66sb"]
Sep 15 21:44:09.902402 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:44:09.902366    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8712887-51a5-4092-9ab6-a4ef8bbac6a5" path="/var/lib/kubelet/pods/b8712887-51a5-4092-9ab6-a4ef8bbac6a5/volumes"
Sep 15 21:45:00.156511 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.156480    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"]
Sep 15 21:45:00.157140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.156988    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8712887-51a5-4092-9ab6-a4ef8bbac6a5" containerName="payload-processing"
Sep 15 21:45:00.157140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.157007    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8712887-51a5-4092-9ab6-a4ef8bbac6a5" containerName="payload-processing"
Sep 15 21:45:00.157140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.157035    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72213fc9-f840-4bfb-b6b5-0f0258b185a2" containerName="maas-api"
Sep 15 21:45:00.157140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.157043    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72213fc9-f840-4bfb-b6b5-0f0258b185a2" containerName="maas-api"
Sep 15 21:45:00.157140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.157139    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72213fc9-f840-4bfb-b6b5-0f0258b185a2" containerName="maas-api"
Sep 15 21:45:00.157305 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.157149    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8712887-51a5-4092-9ab6-a4ef8bbac6a5" containerName="payload-processing"
Sep 15 21:45:00.160621 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.160600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"
Sep 15 21:45:00.163656 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.163635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7fhv\""
Sep 15 21:45:00.169631 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.169600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"]
Sep 15 21:45:00.173065 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.173045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"
Sep 15 21:45:00.452259 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:00.452228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"]
Sep 15 21:45:00.453308 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:45:00.453281    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215be0b5_8455_433b_9bc3_2a98f223073d.slice/crio-bf875938932d868e8d557aa0ee7d6aca7a85aa1a2eda7c8bc0aaf63304e3e9f8 WatchSource:0}: Error finding container bf875938932d868e8d557aa0ee7d6aca7a85aa1a2eda7c8bc0aaf63304e3e9f8: Status 404 returned error can't find the container with id bf875938932d868e8d557aa0ee7d6aca7a85aa1a2eda7c8bc0aaf63304e3e9f8
Sep 15 21:45:01.442927 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:01.442897    2563 generic.go:356] "Generic (PLEG): container finished" podID="215be0b5-8455-433b-9bc3-2a98f223073d" containerID="38475eb7917c9508e1afa230460cf632598f82f05d221da820f71d0de57b8545" exitCode=0
Sep 15 21:45:01.443313 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:01.442975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw" event={"ID":"215be0b5-8455-433b-9bc3-2a98f223073d","Type":"ContainerDied","Data":"38475eb7917c9508e1afa230460cf632598f82f05d221da820f71d0de57b8545"}
Sep 15 21:45:01.443313 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:01.443034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw" event={"ID":"215be0b5-8455-433b-9bc3-2a98f223073d","Type":"ContainerStarted","Data":"bf875938932d868e8d557aa0ee7d6aca7a85aa1a2eda7c8bc0aaf63304e3e9f8"}
Sep 15 21:45:02.588560 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:02.588539    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"
Sep 15 21:45:03.455995 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:03.455959    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"
Sep 15 21:45:03.456168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:03.455963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw" event={"ID":"215be0b5-8455-433b-9bc3-2a98f223073d","Type":"ContainerDied","Data":"bf875938932d868e8d557aa0ee7d6aca7a85aa1a2eda7c8bc0aaf63304e3e9f8"}
Sep 15 21:45:03.456168 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:03.456121    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf875938932d868e8d557aa0ee7d6aca7a85aa1a2eda7c8bc0aaf63304e3e9f8"
Sep 15 21:45:33.262638 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:33.262604    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"]
Sep 15 21:45:33.267101 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:33.267079    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e985b6-29825145-cjmrw"]
Sep 15 21:45:33.902319 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:33.902286    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215be0b5-8455-433b-9bc3-2a98f223073d" path="/var/lib/kubelet/pods/215be0b5-8455-433b-9bc3-2a98f223073d/volumes"
Sep 15 21:45:51.388780 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.388742    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"]
Sep 15 21:45:51.389266 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.389053    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2" podUID="e96950d4-cc38-4001-ac6f-85b9aa344755" containerName="maas-api" containerID="cri-o://9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f" gracePeriod=30
Sep 15 21:45:51.646055 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.645996    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:45:51.711301 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.711265    2563 generic.go:356] "Generic (PLEG): container finished" podID="e96950d4-cc38-4001-ac6f-85b9aa344755" containerID="9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f" exitCode=0
Sep 15 21:45:51.711502 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.711307    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"
Sep 15 21:45:51.711502 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.711312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2" event={"ID":"e96950d4-cc38-4001-ac6f-85b9aa344755","Type":"ContainerDied","Data":"9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f"}
Sep 15 21:45:51.711502 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.711417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2" event={"ID":"e96950d4-cc38-4001-ac6f-85b9aa344755","Type":"ContainerDied","Data":"cef9219d1969e0454d09aeb00c08dad244cc326b9d32a6b177bfc1cfcf6a9feb"}
Sep 15 21:45:51.711502 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.711439    2563 scope.go:117] "RemoveContainer" containerID="9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f"
Sep 15 21:45:51.721359 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.721340    2563 scope.go:117] "RemoveContainer" containerID="9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f"
Sep 15 21:45:51.721715 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:45:51.721691    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f\": container with ID starting with 9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f not found: ID does not exist" containerID="9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f"
Sep 15 21:45:51.721776 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.721725    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f"} err="failed to get container status \"9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f\": rpc error: code = NotFound desc = could not find container \"9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f\": container with ID starting with 9ae4d683233f5168e2c0bde97fbf84691ccb78fd519a241194e70a7ce059233f not found: ID does not exist"
Sep 15 21:45:51.740298 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.740274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96950d4-cc38-4001-ac6f-85b9aa344755-openshift-service-ca\") pod \"e96950d4-cc38-4001-ac6f-85b9aa344755\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") "
Sep 15 21:45:51.740426 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.740313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-metrics-certs\") pod \"e96950d4-cc38-4001-ac6f-85b9aa344755\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") "
Sep 15 21:45:51.740426 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.740371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9vb9\" (UniqueName: \"kubernetes.io/projected/e96950d4-cc38-4001-ac6f-85b9aa344755-kube-api-access-z9vb9\") pod \"e96950d4-cc38-4001-ac6f-85b9aa344755\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") "
Sep 15 21:45:51.740527 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.740470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls\") pod \"e96950d4-cc38-4001-ac6f-85b9aa344755\" (UID: \"e96950d4-cc38-4001-ac6f-85b9aa344755\") "
Sep 15 21:45:51.740785 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.740759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96950d4-cc38-4001-ac6f-85b9aa344755-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e96950d4-cc38-4001-ac6f-85b9aa344755" (UID: "e96950d4-cc38-4001-ac6f-85b9aa344755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:45:51.740901 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.740864    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96950d4-cc38-4001-ac6f-85b9aa344755-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:45:51.742879 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.742850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e96950d4-cc38-4001-ac6f-85b9aa344755" (UID: "e96950d4-cc38-4001-ac6f-85b9aa344755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:45:51.742995 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.742943    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e96950d4-cc38-4001-ac6f-85b9aa344755" (UID: "e96950d4-cc38-4001-ac6f-85b9aa344755"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:45:51.742995 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.742972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96950d4-cc38-4001-ac6f-85b9aa344755-kube-api-access-z9vb9" (OuterVolumeSpecName: "kube-api-access-z9vb9") pod "e96950d4-cc38-4001-ac6f-85b9aa344755" (UID: "e96950d4-cc38-4001-ac6f-85b9aa344755"). InnerVolumeSpecName "kube-api-access-z9vb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:51.841856 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.841815    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:45:51.841856 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.841847    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96950d4-cc38-4001-ac6f-85b9aa344755-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:45:51.841856 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:51.841861    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9vb9\" (UniqueName: \"kubernetes.io/projected/e96950d4-cc38-4001-ac6f-85b9aa344755-kube-api-access-z9vb9\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:45:52.028256 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:52.028222    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"]
Sep 15 21:45:52.030389 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:52.030365    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3c4dc8-7584c6fc9f-n9cd2"]
Sep 15 21:45:53.902608 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:45:53.902574    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96950d4-cc38-4001-ac6f-85b9aa344755" path="/var/lib/kubelet/pods/e96950d4-cc38-4001-ac6f-85b9aa344755/volumes"
Sep 15 21:46:09.392229 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.392188    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"]
Sep 15 21:46:09.393086 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.393031    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7" podUID="b17b4bf0-44d0-418d-9537-02e1690fb98b" containerName="maas-api" containerID="cri-o://bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1" gracePeriod=30
Sep 15 21:46:09.455038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.454950    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"]
Sep 15 21:46:09.455295 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.455214    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj" podUID="301211d7-dedf-4cbe-a2e2-17048319187c" containerName="payload-processing" containerID="cri-o://46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013" gracePeriod=30
Sep 15 21:46:09.736566 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.736542    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:46:09.739665 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.739637    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:46:09.810147 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.810087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5tzn\" (UniqueName: \"kubernetes.io/projected/b17b4bf0-44d0-418d-9537-02e1690fb98b-kube-api-access-l5tzn\") pod \"b17b4bf0-44d0-418d-9537-02e1690fb98b\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") "
Sep 15 21:46:09.810147 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.810123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17b4bf0-44d0-418d-9537-02e1690fb98b-openshift-service-ca\") pod \"b17b4bf0-44d0-418d-9537-02e1690fb98b\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") "
Sep 15 21:46:09.810147 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.810143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-metrics-certs\") pod \"b17b4bf0-44d0-418d-9537-02e1690fb98b\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") "
Sep 15 21:46:09.810846 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.810167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw8kg\" (UniqueName: \"kubernetes.io/projected/301211d7-dedf-4cbe-a2e2-17048319187c-kube-api-access-bw8kg\") pod \"301211d7-dedf-4cbe-a2e2-17048319187c\" (UID: \"301211d7-dedf-4cbe-a2e2-17048319187c\") "
Sep 15 21:46:09.810846 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.810202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls\") pod \"b17b4bf0-44d0-418d-9537-02e1690fb98b\" (UID: \"b17b4bf0-44d0-418d-9537-02e1690fb98b\") "
Sep 15 21:46:09.810846 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.810275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301211d7-dedf-4cbe-a2e2-17048319187c-plugins-config-volume\") pod \"301211d7-dedf-4cbe-a2e2-17048319187c\" (UID: \"301211d7-dedf-4cbe-a2e2-17048319187c\") "
Sep 15 21:46:09.810846 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.810544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17b4bf0-44d0-418d-9537-02e1690fb98b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b17b4bf0-44d0-418d-9537-02e1690fb98b" (UID: "b17b4bf0-44d0-418d-9537-02e1690fb98b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:09.811057 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.810890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301211d7-dedf-4cbe-a2e2-17048319187c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "301211d7-dedf-4cbe-a2e2-17048319187c" (UID: "301211d7-dedf-4cbe-a2e2-17048319187c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:09.812895 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.812866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17b4bf0-44d0-418d-9537-02e1690fb98b-kube-api-access-l5tzn" (OuterVolumeSpecName: "kube-api-access-l5tzn") pod "b17b4bf0-44d0-418d-9537-02e1690fb98b" (UID: "b17b4bf0-44d0-418d-9537-02e1690fb98b"). InnerVolumeSpecName "kube-api-access-l5tzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:09.813019 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.812962    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b17b4bf0-44d0-418d-9537-02e1690fb98b" (UID: "b17b4bf0-44d0-418d-9537-02e1690fb98b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:09.813256 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.813237    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301211d7-dedf-4cbe-a2e2-17048319187c-kube-api-access-bw8kg" (OuterVolumeSpecName: "kube-api-access-bw8kg") pod "301211d7-dedf-4cbe-a2e2-17048319187c" (UID: "301211d7-dedf-4cbe-a2e2-17048319187c"). InnerVolumeSpecName "kube-api-access-bw8kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:09.813321 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.813286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b17b4bf0-44d0-418d-9537-02e1690fb98b" (UID: "b17b4bf0-44d0-418d-9537-02e1690fb98b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:09.817923 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.817900    2563 generic.go:356] "Generic (PLEG): container finished" podID="b17b4bf0-44d0-418d-9537-02e1690fb98b" containerID="bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1" exitCode=0
Sep 15 21:46:09.818038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.817949    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"
Sep 15 21:46:09.818038 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.817980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7" event={"ID":"b17b4bf0-44d0-418d-9537-02e1690fb98b","Type":"ContainerDied","Data":"bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1"}
Sep 15 21:46:09.818139 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.818040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7" event={"ID":"b17b4bf0-44d0-418d-9537-02e1690fb98b","Type":"ContainerDied","Data":"8b580be3a4ded41a14e5201a54848f671c77c5e3db58f7dd29102438dca7710c"}
Sep 15 21:46:09.818139 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.818071    2563 scope.go:117] "RemoveContainer" containerID="bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1"
Sep 15 21:46:09.820140 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.820118    2563 generic.go:356] "Generic (PLEG): container finished" podID="301211d7-dedf-4cbe-a2e2-17048319187c" containerID="46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013" exitCode=0
Sep 15 21:46:09.820244 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.820163    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"
Sep 15 21:46:09.820244 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.820198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj" event={"ID":"301211d7-dedf-4cbe-a2e2-17048319187c","Type":"ContainerDied","Data":"46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013"}
Sep 15 21:46:09.820244 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.820243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj" event={"ID":"301211d7-dedf-4cbe-a2e2-17048319187c","Type":"ContainerDied","Data":"a2707bcc76bc1c4daf7fca5431bf4631db47989f90c2aec00269e9e4b1004745"}
Sep 15 21:46:09.830348 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.830331    2563 scope.go:117] "RemoveContainer" containerID="bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1"
Sep 15 21:46:09.830837 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:46:09.830690    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1\": container with ID starting with bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1 not found: ID does not exist" containerID="bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1"
Sep 15 21:46:09.830837 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.830730    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1"} err="failed to get container status \"bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1\": rpc error: code = NotFound desc = could not find container \"bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1\": container with ID starting with bd71add06644454de7ce7405dc66ec667eeb12a94c4e65cd5f34c269c88aa1a1 not found: ID does not exist"
Sep 15 21:46:09.830837 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.830754    2563 scope.go:117] "RemoveContainer" containerID="46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013"
Sep 15 21:46:09.839087 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.839070    2563 scope.go:117] "RemoveContainer" containerID="46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013"
Sep 15 21:46:09.839415 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:46:09.839390    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013\": container with ID starting with 46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013 not found: ID does not exist" containerID="46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013"
Sep 15 21:46:09.839545 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.839415    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013"} err="failed to get container status \"46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013\": rpc error: code = NotFound desc = could not find container \"46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013\": container with ID starting with 46f05a2548363fa2d8d479ea1677a284dfda2a5271a3dc63e9981eecabe20013 not found: ID does not exist"
Sep 15 21:46:09.842102 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.842082    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"]
Sep 15 21:46:09.848627 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.848598    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7c57b6-8f894cb65-rm9t7"]
Sep 15 21:46:09.859543 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.859522    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"]
Sep 15 21:46:09.863662 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.863641    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7c57b6-84fc59d6c7-6gktj"]
Sep 15 21:46:09.901857 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.901827    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="301211d7-dedf-4cbe-a2e2-17048319187c" path="/var/lib/kubelet/pods/301211d7-dedf-4cbe-a2e2-17048319187c/volumes"
Sep 15 21:46:09.902215 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.902200    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b17b4bf0-44d0-418d-9537-02e1690fb98b" path="/var/lib/kubelet/pods/b17b4bf0-44d0-418d-9537-02e1690fb98b/volumes"
Sep 15 21:46:09.912182 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.912147    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301211d7-dedf-4cbe-a2e2-17048319187c-plugins-config-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:09.912377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.912343    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5tzn\" (UniqueName: \"kubernetes.io/projected/b17b4bf0-44d0-418d-9537-02e1690fb98b-kube-api-access-l5tzn\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:09.912377 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.912370    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17b4bf0-44d0-418d-9537-02e1690fb98b-openshift-service-ca\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:09.912621 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.912384    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:09.912621 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.912399    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8kg\" (UniqueName: \"kubernetes.io/projected/301211d7-dedf-4cbe-a2e2-17048319187c-kube-api-access-bw8kg\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:09.912621 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:09.912412    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b17b4bf0-44d0-418d-9537-02e1690fb98b-maas-api-tls\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:19.344600 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.344563    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b5854d44-5ll6l"]
Sep 15 21:46:19.345205 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.344862    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65b5854d44-5ll6l" podUID="aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" containerName="manager" containerID="cri-o://963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa" gracePeriod=30
Sep 15 21:46:19.582800 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.582776    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:46:19.699425 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.699342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l947b\" (UniqueName: \"kubernetes.io/projected/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-kube-api-access-l947b\") pod \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") "
Sep 15 21:46:19.699612 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.699467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-webhook-cert\") pod \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") "
Sep 15 21:46:19.699612 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.699531    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-metrics-certs\") pod \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\" (UID: \"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2\") "
Sep 15 21:46:19.701755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.701727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" (UID: "aa60b90f-dc4c-4bfb-9f7f-eed152473ba2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:19.701867 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.701757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-kube-api-access-l947b" (OuterVolumeSpecName: "kube-api-access-l947b") pod "aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" (UID: "aa60b90f-dc4c-4bfb-9f7f-eed152473ba2"). InnerVolumeSpecName "kube-api-access-l947b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:19.701867 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.701807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" (UID: "aa60b90f-dc4c-4bfb-9f7f-eed152473ba2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:19.800267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.800239    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-webhook-cert\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:19.800267 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.800263    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:19.800473 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.800274    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l947b\" (UniqueName: \"kubernetes.io/projected/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2-kube-api-access-l947b\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:46:19.874486 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.874436    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" containerID="963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa" exitCode=0
Sep 15 21:46:19.874676 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.874486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-5ll6l" event={"ID":"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2","Type":"ContainerDied","Data":"963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa"}
Sep 15 21:46:19.874676 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.874501    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-5ll6l"
Sep 15 21:46:19.874676 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.874542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-5ll6l" event={"ID":"aa60b90f-dc4c-4bfb-9f7f-eed152473ba2","Type":"ContainerDied","Data":"bc6cf421ae44f5513212f0c6ef738b7adc2731eace85a945a3f1a8894b0694dc"}
Sep 15 21:46:19.874676 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.874565    2563 scope.go:117] "RemoveContainer" containerID="963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa"
Sep 15 21:46:19.885114 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.885098    2563 scope.go:117] "RemoveContainer" containerID="963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa"
Sep 15 21:46:19.885369 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:46:19.885351    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa\": container with ID starting with 963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa not found: ID does not exist" containerID="963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa"
Sep 15 21:46:19.885428 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.885375    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa"} err="failed to get container status \"963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa\": rpc error: code = NotFound desc = could not find container \"963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa\": container with ID starting with 963b8b017566d79ff0b03e90cfadbf0319626c4a1bce59839efe7e8b52a849aa not found: ID does not exist"
Sep 15 21:46:19.901630 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.901607    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b5854d44-5ll6l"]
Sep 15 21:46:19.903802 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:19.903780    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65b5854d44-5ll6l"]
Sep 15 21:46:21.900866 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:46:21.900825    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" path="/var/lib/kubelet/pods/aa60b90f-dc4c-4bfb-9f7f-eed152473ba2/volumes"
Sep 15 21:47:20.750490 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750437    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65b5854d44-s2xkk"]
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750839    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301211d7-dedf-4cbe-a2e2-17048319187c" containerName="payload-processing"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750851    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="301211d7-dedf-4cbe-a2e2-17048319187c" containerName="payload-processing"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750859    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215be0b5-8455-433b-9bc3-2a98f223073d" containerName="cleanup"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750864    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="215be0b5-8455-433b-9bc3-2a98f223073d" containerName="cleanup"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750874    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17b4bf0-44d0-418d-9537-02e1690fb98b" containerName="maas-api"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17b4bf0-44d0-418d-9537-02e1690fb98b" containerName="maas-api"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750893    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" containerName="manager"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750898    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" containerName="manager"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750904    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96950d4-cc38-4001-ac6f-85b9aa344755" containerName="maas-api"
Sep 15 21:47:20.750912 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750909    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96950d4-cc38-4001-ac6f-85b9aa344755" containerName="maas-api"
Sep 15 21:47:20.751345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750977    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b17b4bf0-44d0-418d-9537-02e1690fb98b" containerName="maas-api"
Sep 15 21:47:20.751345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750985    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="301211d7-dedf-4cbe-a2e2-17048319187c" containerName="payload-processing"
Sep 15 21:47:20.751345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750991    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96950d4-cc38-4001-ac6f-85b9aa344755" containerName="maas-api"
Sep 15 21:47:20.751345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.750999    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa60b90f-dc4c-4bfb-9f7f-eed152473ba2" containerName="manager"
Sep 15 21:47:20.751345 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.751006    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="215be0b5-8455-433b-9bc3-2a98f223073d" containerName="cleanup"
Sep 15 21:47:20.754226 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.754210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.757024 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.757003    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:47:20.757129 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.757040    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:47:20.757129 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.757048    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:47:20.757129 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.757050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wvswp\""
Sep 15 21:47:20.765721 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.765693    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b5854d44-s2xkk"]
Sep 15 21:47:20.836035 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.836001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-metrics-certs\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.836191 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.836078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-webhook-cert\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.836191 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.836139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ssrw\" (UniqueName: \"kubernetes.io/projected/c112cede-1418-4dee-ad3b-b1913047f0ca-kube-api-access-8ssrw\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.937383 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.937347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-webhook-cert\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.937554 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.937406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssrw\" (UniqueName: \"kubernetes.io/projected/c112cede-1418-4dee-ad3b-b1913047f0ca-kube-api-access-8ssrw\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.937554 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.937441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-metrics-certs\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.940094 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.940055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-metrics-certs\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.940094 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.940079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-webhook-cert\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:20.945011 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:20.944993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ssrw\" (UniqueName: \"kubernetes.io/projected/c112cede-1418-4dee-ad3b-b1913047f0ca-kube-api-access-8ssrw\") pod \"maas-controller-65b5854d44-s2xkk\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") " pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:21.065285 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:21.065202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:21.336303 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:21.336279    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b5854d44-s2xkk"]
Sep 15 21:47:21.338794 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:47:21.338763    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc112cede_1418_4dee_ad3b_b1913047f0ca.slice/crio-e08db8475f4a4d861323871bfd3e7b7a228763cc56bde90a773acf392f010c32 WatchSource:0}: Error finding container e08db8475f4a4d861323871bfd3e7b7a228763cc56bde90a773acf392f010c32: Status 404 returned error can't find the container with id e08db8475f4a4d861323871bfd3e7b7a228763cc56bde90a773acf392f010c32
Sep 15 21:47:22.193127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:22.193038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-s2xkk" event={"ID":"c112cede-1418-4dee-ad3b-b1913047f0ca","Type":"ContainerStarted","Data":"e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21"}
Sep 15 21:47:22.193127 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:22.193095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-s2xkk" event={"ID":"c112cede-1418-4dee-ad3b-b1913047f0ca","Type":"ContainerStarted","Data":"e08db8475f4a4d861323871bfd3e7b7a228763cc56bde90a773acf392f010c32"}
Sep 15 21:47:22.193562 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:22.193172    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:22.210936 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:22.210887    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65b5854d44-s2xkk" podStartSLOduration=1.698393952 podStartE2EDuration="2.210870848s" totalImagesPullingTime="512.476896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:22.209076001 +0000 UTC m=+2360.808988111" watchObservedRunningTime="2026-09-15 21:47:22.210870848 +0000 UTC m=+2360.810782958"
Sep 15 21:47:33.201028 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:33.200993    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:40.492669 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.492627    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b5854d44-s2xkk"]
Sep 15 21:47:40.493251 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.492906    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65b5854d44-s2xkk" podUID="c112cede-1418-4dee-ad3b-b1913047f0ca" containerName="manager" containerID="cri-o://e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21" gracePeriod=30
Sep 15 21:47:40.741005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.740982    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:40.826304 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.826218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ssrw\" (UniqueName: \"kubernetes.io/projected/c112cede-1418-4dee-ad3b-b1913047f0ca-kube-api-access-8ssrw\") pod \"c112cede-1418-4dee-ad3b-b1913047f0ca\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") "
Sep 15 21:47:40.826304 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.826293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-webhook-cert\") pod \"c112cede-1418-4dee-ad3b-b1913047f0ca\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") "
Sep 15 21:47:40.826530 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.826387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-metrics-certs\") pod \"c112cede-1418-4dee-ad3b-b1913047f0ca\" (UID: \"c112cede-1418-4dee-ad3b-b1913047f0ca\") "
Sep 15 21:47:40.828599 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.828570    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c112cede-1418-4dee-ad3b-b1913047f0ca" (UID: "c112cede-1418-4dee-ad3b-b1913047f0ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:47:40.828722 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.828601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c112cede-1418-4dee-ad3b-b1913047f0ca-kube-api-access-8ssrw" (OuterVolumeSpecName: "kube-api-access-8ssrw") pod "c112cede-1418-4dee-ad3b-b1913047f0ca" (UID: "c112cede-1418-4dee-ad3b-b1913047f0ca"). InnerVolumeSpecName "kube-api-access-8ssrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:47:40.828722 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.828654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c112cede-1418-4dee-ad3b-b1913047f0ca" (UID: "c112cede-1418-4dee-ad3b-b1913047f0ca"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:47:40.928250 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.928214    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-metrics-certs\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:47:40.928250 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.928250    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ssrw\" (UniqueName: \"kubernetes.io/projected/c112cede-1418-4dee-ad3b-b1913047f0ca-kube-api-access-8ssrw\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:47:40.928584 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:40.928264    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c112cede-1418-4dee-ad3b-b1913047f0ca-webhook-cert\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:47:41.299291 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.299253    2563 generic.go:356] "Generic (PLEG): container finished" podID="c112cede-1418-4dee-ad3b-b1913047f0ca" containerID="e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21" exitCode=0
Sep 15 21:47:41.299291 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.299294    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-s2xkk"
Sep 15 21:47:41.299591 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.299338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-s2xkk" event={"ID":"c112cede-1418-4dee-ad3b-b1913047f0ca","Type":"ContainerDied","Data":"e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21"}
Sep 15 21:47:41.299591 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.299397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-s2xkk" event={"ID":"c112cede-1418-4dee-ad3b-b1913047f0ca","Type":"ContainerDied","Data":"e08db8475f4a4d861323871bfd3e7b7a228763cc56bde90a773acf392f010c32"}
Sep 15 21:47:41.299591 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.299425    2563 scope.go:117] "RemoveContainer" containerID="e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21"
Sep 15 21:47:41.309211 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.309193    2563 scope.go:117] "RemoveContainer" containerID="e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21"
Sep 15 21:47:41.309530 ip-10-0-140-113 kubenswrapper[2563]: E0915 21:47:41.309507    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21\": container with ID starting with e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21 not found: ID does not exist" containerID="e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21"
Sep 15 21:47:41.309614 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.309538    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21"} err="failed to get container status \"e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21\": rpc error: code = NotFound desc = could not find container \"e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21\": container with ID starting with e05567f7a625bb0f3799db3a9fe4066863e381665f804bee800fd9cf1edb4c21 not found: ID does not exist"
Sep 15 21:47:41.319327 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.319302    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b5854d44-s2xkk"]
Sep 15 21:47:41.323350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.323331    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65b5854d44-s2xkk"]
Sep 15 21:47:41.903776 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:47:41.903745    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c112cede-1418-4dee-ad3b-b1913047f0ca" path="/var/lib/kubelet/pods/c112cede-1418-4dee-ad3b-b1913047f0ca/volumes"
Sep 15 21:49:02.464137 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:02.464101    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"]
Sep 15 21:49:02.464755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:02.464344    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6" podUID="cf6c48f7-b36c-4226-9105-848931709f56" containerName="manager" containerID="cri-o://d6c66a1333c0ad7dda85513c9c4bd31d581401ec5bca87257d15e062f8f8eeec" gracePeriod=10
Sep 15 21:49:02.723827 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:02.723728    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf6c48f7-b36c-4226-9105-848931709f56" containerID="d6c66a1333c0ad7dda85513c9c4bd31d581401ec5bca87257d15e062f8f8eeec" exitCode=0
Sep 15 21:49:02.723827 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:02.723810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6" event={"ID":"cf6c48f7-b36c-4226-9105-848931709f56","Type":"ContainerDied","Data":"d6c66a1333c0ad7dda85513c9c4bd31d581401ec5bca87257d15e062f8f8eeec"}
Sep 15 21:49:03.234991 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.234960    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:49:03.372212 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.372103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28zps\" (UniqueName: \"kubernetes.io/projected/cf6c48f7-b36c-4226-9105-848931709f56-kube-api-access-28zps\") pod \"cf6c48f7-b36c-4226-9105-848931709f56\" (UID: \"cf6c48f7-b36c-4226-9105-848931709f56\") "
Sep 15 21:49:03.372212 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.372197    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf6c48f7-b36c-4226-9105-848931709f56-extensions-socket-volume\") pod \"cf6c48f7-b36c-4226-9105-848931709f56\" (UID: \"cf6c48f7-b36c-4226-9105-848931709f56\") "
Sep 15 21:49:03.372661 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.372635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6c48f7-b36c-4226-9105-848931709f56-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cf6c48f7-b36c-4226-9105-848931709f56" (UID: "cf6c48f7-b36c-4226-9105-848931709f56"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:49:03.374550 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.374524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6c48f7-b36c-4226-9105-848931709f56-kube-api-access-28zps" (OuterVolumeSpecName: "kube-api-access-28zps") pod "cf6c48f7-b36c-4226-9105-848931709f56" (UID: "cf6c48f7-b36c-4226-9105-848931709f56"). InnerVolumeSpecName "kube-api-access-28zps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:49:03.473463 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.473407    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zps\" (UniqueName: \"kubernetes.io/projected/cf6c48f7-b36c-4226-9105-848931709f56-kube-api-access-28zps\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:49:03.473899 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.473486    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf6c48f7-b36c-4226-9105-848931709f56-extensions-socket-volume\") on node \"ip-10-0-140-113.ec2.internal\" DevicePath \"\""
Sep 15 21:49:03.730030 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.730000    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"
Sep 15 21:49:03.730224 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.730043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6" event={"ID":"cf6c48f7-b36c-4226-9105-848931709f56","Type":"ContainerDied","Data":"b8c77ca5e59d9985ec640efae4334e314ae70e7e117f8a3155ce68993dde34de"}
Sep 15 21:49:03.730224 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.730107    2563 scope.go:117] "RemoveContainer" containerID="d6c66a1333c0ad7dda85513c9c4bd31d581401ec5bca87257d15e062f8f8eeec"
Sep 15 21:49:03.752474 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.752404    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"]
Sep 15 21:49:03.760258 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.760207    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pbhx6"]
Sep 15 21:49:03.906770 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:49:03.906734    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf6c48f7-b36c-4226-9105-848931709f56" path="/var/lib/kubelet/pods/cf6c48f7-b36c-4226-9105-848931709f56/volumes"
Sep 15 21:50:03.156466 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.156419    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"]
Sep 15 21:50:03.156885 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.156868    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c112cede-1418-4dee-ad3b-b1913047f0ca" containerName="manager"
Sep 15 21:50:03.156932 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.156891    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c112cede-1418-4dee-ad3b-b1913047f0ca" containerName="manager"
Sep 15 21:50:03.156932 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.156911    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6c48f7-b36c-4226-9105-848931709f56" containerName="manager"
Sep 15 21:50:03.156932 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.156916    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6c48f7-b36c-4226-9105-848931709f56" containerName="manager"
Sep 15 21:50:03.157029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.156990    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6c48f7-b36c-4226-9105-848931709f56" containerName="manager"
Sep 15 21:50:03.157029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.157003    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c112cede-1418-4dee-ad3b-b1913047f0ca" containerName="manager"
Sep 15 21:50:03.161264 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.161247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:03.163572 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.163550    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qrkx8\""
Sep 15 21:50:03.178285 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.178253    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"]
Sep 15 21:50:03.204922 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.204893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrl2k\" (UniqueName: \"kubernetes.io/projected/94aec559-fbc0-4bb2-b40f-144786ef6fed-kube-api-access-mrl2k\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hp8xt\" (UID: \"94aec559-fbc0-4bb2-b40f-144786ef6fed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:03.205040 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.204941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94aec559-fbc0-4bb2-b40f-144786ef6fed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hp8xt\" (UID: \"94aec559-fbc0-4bb2-b40f-144786ef6fed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:03.306350 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.306315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrl2k\" (UniqueName: \"kubernetes.io/projected/94aec559-fbc0-4bb2-b40f-144786ef6fed-kube-api-access-mrl2k\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hp8xt\" (UID: \"94aec559-fbc0-4bb2-b40f-144786ef6fed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:03.306553 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.306372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94aec559-fbc0-4bb2-b40f-144786ef6fed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hp8xt\" (UID: \"94aec559-fbc0-4bb2-b40f-144786ef6fed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:03.306705 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.306687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94aec559-fbc0-4bb2-b40f-144786ef6fed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hp8xt\" (UID: \"94aec559-fbc0-4bb2-b40f-144786ef6fed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:03.316235 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.316217    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrl2k\" (UniqueName: \"kubernetes.io/projected/94aec559-fbc0-4bb2-b40f-144786ef6fed-kube-api-access-mrl2k\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hp8xt\" (UID: \"94aec559-fbc0-4bb2-b40f-144786ef6fed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:03.471678 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.471650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:03.757855 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:03.757828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"]
Sep 15 21:50:03.760042 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:50:03.760010    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94aec559_fbc0_4bb2_b40f_144786ef6fed.slice/crio-b49b888b9a27dd93f76e849fe7211a31cd6421179b20a8e8d84c4a486f459af3 WatchSource:0}: Error finding container b49b888b9a27dd93f76e849fe7211a31cd6421179b20a8e8d84c4a486f459af3: Status 404 returned error can't find the container with id b49b888b9a27dd93f76e849fe7211a31cd6421179b20a8e8d84c4a486f459af3
Sep 15 21:50:04.039910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:04.039818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt" event={"ID":"94aec559-fbc0-4bb2-b40f-144786ef6fed","Type":"ContainerStarted","Data":"0648699fe14905a0437b0ac20def50e2a5614c0d86c7d162085a52c0c71075cb"}
Sep 15 21:50:04.039910 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:04.039874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt" event={"ID":"94aec559-fbc0-4bb2-b40f-144786ef6fed","Type":"ContainerStarted","Data":"b49b888b9a27dd93f76e849fe7211a31cd6421179b20a8e8d84c4a486f459af3"}
Sep 15 21:50:04.040099 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:04.039956    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:50:04.063390 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:04.063342    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt" podStartSLOduration=1.063327364 podStartE2EDuration="1.063327364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:50:04.061051927 +0000 UTC m=+2522.660964038" watchObservedRunningTime="2026-09-15 21:50:04.063327364 +0000 UTC m=+2522.663239473"
Sep 15 21:50:15.046755 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:50:15.046720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hp8xt"
Sep 15 21:51:02.321778 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:51:02.321743    2563 scope.go:117] "RemoveContainer" containerID="38475eb7917c9508e1afa230460cf632598f82f05d221da820f71d0de57b8545"
Sep 15 21:52:15.975534 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:15.975502    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546ffd48db-59vbp_1588d5a6-75ff-4a81-adf0-3d94cebe87d6/copy-manifests/0.log"
Sep 15 21:52:15.982787 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:15.982754    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546ffd48db-59vbp_1588d5a6-75ff-4a81-adf0-3d94cebe87d6/manager/0.log"
Sep 15 21:52:17.879952 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:17.879914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-cbcv7_933f7889-b755-4c04-912b-faf8ea7c52be/manager/0.log"
Sep 15 21:52:18.235040 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:18.235005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hp8xt_94aec559-fbc0-4bb2-b40f-144786ef6fed/manager/0.log"
Sep 15 21:52:18.470029 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:18.470000    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-tncjr_37953e7e-906e-45b2-9087-2d143b8f7487/manager/0.log"
Sep 15 21:52:18.949757 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:18.949728    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8rwkn_f9d540ad-1119-4ee4-bdce-a236e94c6ffe/discovery/0.log"
Sep 15 21:52:19.065269 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:19.065239    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-695d9b6867-77rmc_1175e460-a0fb-448a-b69c-0566d4f36878/kube-auth-proxy/0.log"
Sep 15 21:52:19.423216 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:19.423189    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6zrlv_c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766/payload-pre-processing/0.log"
Sep 15 21:52:19.540125 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:19.540083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fsqhb_023c1a26-91e1-4d9a-8871-fe737cdac0c5/payload-processing/0.log"
Sep 15 21:52:20.092822 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:20.092788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-876k2_9e1416c9-a759-4821-8323-4049dad5b202/main/0.log"
Sep 15 21:52:20.317877 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:20.317832    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-t7wzf_e2c4caac-9cc7-4c73-b13d-a0003401e0a1/main/0.log"
Sep 15 21:52:20.533489 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:20.533419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vcx9v_1e5e7642-1bcc-41b5-9c17-c30fe08e026f/main/0.log"
Sep 15 21:52:28.929758 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:28.929722    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tcvp4_a12dd506-758c-4108-a967-70c0c1220114/global-pull-secret-syncer/0.log"
Sep 15 21:52:29.060967 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:29.060915    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xthx4_a3e119ff-4a7f-4b5e-aafe-c48b610649af/konnectivity-agent/0.log"
Sep 15 21:52:29.113158 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:29.113131    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-113.ec2.internal_ab1af32928d1c75375f28ea688cf59c1/haproxy/0.log"
Sep 15 21:52:33.565934 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:33.565905    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-cbcv7_933f7889-b755-4c04-912b-faf8ea7c52be/manager/0.log"
Sep 15 21:52:33.758994 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:33.758968    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hp8xt_94aec559-fbc0-4bb2-b40f-144786ef6fed/manager/0.log"
Sep 15 21:52:33.908386 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:33.908304    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-tncjr_37953e7e-906e-45b2-9087-2d143b8f7487/manager/0.log"
Sep 15 21:52:35.661167 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:35.661139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-5vx6t_2553f442-cb63-4da5-9a2c-eeb5ba3db7d0/cluster-monitoring-operator/0.log"
Sep 15 21:52:35.920782 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:35.920752    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85cf98b868-7brsf_f280aa87-d313-4477-9a52-8635473c5956/metrics-server/0.log"
Sep 15 21:52:36.043788 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.043683    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtgmx_fa32c370-5087-4f62-9842-10a68b16b8e6/node-exporter/0.log"
Sep 15 21:52:36.105387 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.105311    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtgmx_fa32c370-5087-4f62-9842-10a68b16b8e6/kube-rbac-proxy/0.log"
Sep 15 21:52:36.142750 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.142720    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtgmx_fa32c370-5087-4f62-9842-10a68b16b8e6/init-textfile/0.log"
Sep 15 21:52:36.375462 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.375365    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-p528w_fcb21e97-b29e-4372-9bc0-90e968051901/kube-rbac-proxy-main/0.log"
Sep 15 21:52:36.400608 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.400580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-p528w_fcb21e97-b29e-4372-9bc0-90e968051901/kube-rbac-proxy-self/0.log"
Sep 15 21:52:36.426081 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.426061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-p528w_fcb21e97-b29e-4372-9bc0-90e968051901/openshift-state-metrics/0.log"
Sep 15 21:52:36.483958 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.483937    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_feaf4686-0714-4b1c-934f-569598a45200/prometheus/0.log"
Sep 15 21:52:36.514924 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.514896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_feaf4686-0714-4b1c-934f-569598a45200/config-reloader/0.log"
Sep 15 21:52:36.555045 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.555025    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_feaf4686-0714-4b1c-934f-569598a45200/thanos-sidecar/0.log"
Sep 15 21:52:36.586282 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.586262    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_feaf4686-0714-4b1c-934f-569598a45200/kube-rbac-proxy-web/0.log"
Sep 15 21:52:36.651667 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.651600    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_feaf4686-0714-4b1c-934f-569598a45200/kube-rbac-proxy/0.log"
Sep 15 21:52:36.682925 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.682906    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_feaf4686-0714-4b1c-934f-569598a45200/kube-rbac-proxy-thanos/0.log"
Sep 15 21:52:36.732405 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.732381    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_feaf4686-0714-4b1c-934f-569598a45200/init-config-reloader/0.log"
Sep 15 21:52:36.785595 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.785569    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-n89nc_46bc392e-0b09-4237-9d53-295ead8ef704/prometheus-operator/0.log"
Sep 15 21:52:36.813815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.813796    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-n89nc_46bc392e-0b09-4237-9d53-295ead8ef704/kube-rbac-proxy/0.log"
Sep 15 21:52:36.856218 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:36.856188    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-dlcdn_3a8dacef-1ae1-423c-95cf-148348c154e2/prometheus-operator-admission-webhook/0.log"
Sep 15 21:52:37.033636 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:37.033612    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8656b74474-bnljb_ecddf7c2-12d0-4c33-9b08-6ed39bd26c80/thanos-query/0.log"
Sep 15 21:52:37.070653 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:37.070627    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8656b74474-bnljb_ecddf7c2-12d0-4c33-9b08-6ed39bd26c80/kube-rbac-proxy-web/0.log"
Sep 15 21:52:37.105647 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:37.105617    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8656b74474-bnljb_ecddf7c2-12d0-4c33-9b08-6ed39bd26c80/kube-rbac-proxy/0.log"
Sep 15 21:52:37.150369 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:37.150343    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8656b74474-bnljb_ecddf7c2-12d0-4c33-9b08-6ed39bd26c80/prom-label-proxy/0.log"
Sep 15 21:52:37.187374 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:37.187340    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8656b74474-bnljb_ecddf7c2-12d0-4c33-9b08-6ed39bd26c80/kube-rbac-proxy-rules/0.log"
Sep 15 21:52:37.215641 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:37.215619    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8656b74474-bnljb_ecddf7c2-12d0-4c33-9b08-6ed39bd26c80/kube-rbac-proxy-metrics/0.log"
Sep 15 21:52:38.398035 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.398009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-jq6hz_41a05419-d00c-4312-bf2b-32c368d48def/networking-console-plugin/0.log"
Sep 15 21:52:38.693669 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.693593    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"]
Sep 15 21:52:38.697615 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.697595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.699759 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.699738    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggc6\"/\"openshift-service-ca.crt\""
Sep 15 21:52:38.699914 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.699795    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggc6\"/\"kube-root-ca.crt\""
Sep 15 21:52:38.699989 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.699974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sggc6\"/\"default-dockercfg-zzzv5\""
Sep 15 21:52:38.707427 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.707403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"]
Sep 15 21:52:38.728870 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.728846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6gs\" (UniqueName: \"kubernetes.io/projected/11234597-92fe-4d60-91f9-a8337eb0b877-kube-api-access-tg6gs\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.728976 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.728909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-sys\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.728976 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.728951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-podres\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.728976 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.728968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-lib-modules\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.729080 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.729005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-proc\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.829665 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6gs\" (UniqueName: \"kubernetes.io/projected/11234597-92fe-4d60-91f9-a8337eb0b877-kube-api-access-tg6gs\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.829849 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-sys\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.829849 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-podres\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.829849 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-lib-modules\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.830000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-proc\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.830000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-proc\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.830000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-podres\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.830000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-lib-modules\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.830000 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.829848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11234597-92fe-4d60-91f9-a8337eb0b877-sys\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:38.837880 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:38.837848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6gs\" (UniqueName: \"kubernetes.io/projected/11234597-92fe-4d60-91f9-a8337eb0b877-kube-api-access-tg6gs\") pod \"perf-node-gather-daemonset-fbldq\" (UID: \"11234597-92fe-4d60-91f9-a8337eb0b877\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:39.008351 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:39.008322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:39.292085 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:39.292057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"]
Sep 15 21:52:39.294476 ip-10-0-140-113 kubenswrapper[2563]: W0915 21:52:39.294424    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod11234597_92fe_4d60_91f9_a8337eb0b877.slice/crio-fc769933517d3ec383e76bf6be2204f2223fd619e2943c7ecc7a3b9dafc64cfe WatchSource:0}: Error finding container fc769933517d3ec383e76bf6be2204f2223fd619e2943c7ecc7a3b9dafc64cfe: Status 404 returned error can't find the container with id fc769933517d3ec383e76bf6be2204f2223fd619e2943c7ecc7a3b9dafc64cfe
Sep 15 21:52:39.829602 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:39.829562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq" event={"ID":"11234597-92fe-4d60-91f9-a8337eb0b877","Type":"ContainerStarted","Data":"35d8658bd92a0f2c4239a1ce422856a3e41e9af9035bb93f9635c754454a02d8"}
Sep 15 21:52:39.830023 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:39.829627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq" event={"ID":"11234597-92fe-4d60-91f9-a8337eb0b877","Type":"ContainerStarted","Data":"fc769933517d3ec383e76bf6be2204f2223fd619e2943c7ecc7a3b9dafc64cfe"}
Sep 15 21:52:39.830023 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:39.829654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:39.846075 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:39.846027    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq" podStartSLOduration=1.846011319 podStartE2EDuration="1.846011319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:39.844079751 +0000 UTC m=+2678.443991861" watchObservedRunningTime="2026-09-15 21:52:39.846011319 +0000 UTC m=+2678.445923428"
Sep 15 21:52:40.357402 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:40.357373    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-brljx_95d5ba47-d85a-4922-bf29-9e1eedab4cd7/volume-data-source-validator/0.log"
Sep 15 21:52:41.401795 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:41.401768    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dlqdm_1fc041fc-3382-4159-b3be-8145231efc81/dns/0.log"
Sep 15 21:52:41.424300 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:41.424271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dlqdm_1fc041fc-3382-4159-b3be-8145231efc81/kube-rbac-proxy/0.log"
Sep 15 21:52:41.541715 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:41.541684    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zngks_00f0d21f-f617-40af-998d-895e688399ce/dns-node-resolver/0.log"
Sep 15 21:52:42.097815 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:42.097792    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lmfnh_fd84df4b-1e3b-4784-a5b7-c2ea08ab8877/node-ca/0.log"
Sep 15 21:52:43.892757 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:43.892726    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8rwkn_f9d540ad-1119-4ee4-bdce-a236e94c6ffe/discovery/0.log"
Sep 15 21:52:43.916141 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:43.916111    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-695d9b6867-77rmc_1175e460-a0fb-448a-b69c-0566d4f36878/kube-auth-proxy/0.log"
Sep 15 21:52:44.020897 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:44.020871    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6zrlv_c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766/payload-pre-processing/0.log"
Sep 15 21:52:44.060323 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:44.060297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fsqhb_023c1a26-91e1-4d9a-8871-fe737cdac0c5/payload-processing/0.log"
Sep 15 21:52:45.031691 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:45.031662    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kv4ww_83cfd1f3-a4dc-4534-9151-0640d5a67623/serve-healthcheck-canary/0.log"
Sep 15 21:52:45.551416 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:45.551373    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9299p_7304d05c-061a-4692-89de-6e0fbdad7008/kube-rbac-proxy/0.log"
Sep 15 21:52:45.582005 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:45.581972    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9299p_7304d05c-061a-4692-89de-6e0fbdad7008/exporter/0.log"
Sep 15 21:52:45.609785 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:45.609764    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9299p_7304d05c-061a-4692-89de-6e0fbdad7008/extractor/0.log"
Sep 15 21:52:45.844944 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:45.844867    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-fbldq"
Sep 15 21:52:48.634960 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:48.634931    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546ffd48db-59vbp_1588d5a6-75ff-4a81-adf0-3d94cebe87d6/manager/0.log"
Sep 15 21:52:48.672124 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:48.672093    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546ffd48db-59vbp_1588d5a6-75ff-4a81-adf0-3d94cebe87d6/copy-manifests/0.log"
Sep 15 21:52:49.290851 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:49.290817    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8rwkn_f9d540ad-1119-4ee4-bdce-a236e94c6ffe/discovery/0.log"
Sep 15 21:52:49.304466 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:49.304426    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-695d9b6867-77rmc_1175e460-a0fb-448a-b69c-0566d4f36878/kube-auth-proxy/0.log"
Sep 15 21:52:49.382632 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:49.382593    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6zrlv_c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766/payload-pre-processing/0.log"
Sep 15 21:52:49.418461 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:49.418432    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fsqhb_023c1a26-91e1-4d9a-8871-fe737cdac0c5/payload-processing/0.log"
Sep 15 21:52:50.164656 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:50.164577    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kzszk_d87bda14-7549-4811-9a53-e67940182bc2/openshift-lws-operator/0.log"
Sep 15 21:52:51.827983 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:51.827948    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8rwkn_f9d540ad-1119-4ee4-bdce-a236e94c6ffe/discovery/0.log"
Sep 15 21:52:51.857768 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:51.857740    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-695d9b6867-77rmc_1175e460-a0fb-448a-b69c-0566d4f36878/kube-auth-proxy/0.log"
Sep 15 21:52:51.925520 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:51.925492    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6zrlv_c80ba6c3-fd8e-4df1-bb8e-65ffcb47d766/payload-pre-processing/0.log"
Sep 15 21:52:51.958053 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:51.958025    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fsqhb_023c1a26-91e1-4d9a-8871-fe737cdac0c5/payload-processing/0.log"
Sep 15 21:52:54.916132 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:54.916059    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-brk22_fc04d009-45a3-42b9-9ada-6e5e7458c78b/migrator/0.log"
Sep 15 21:52:54.936606 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:54.936586    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-brk22_fc04d009-45a3-42b9-9ada-6e5e7458c78b/graceful-termination/0.log"
Sep 15 21:52:55.290620 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:55.290587    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-kj6pj_99a17472-2d2b-45ac-98c1-cce356627d5c/kube-storage-version-migrator-operator/1.log"
Sep 15 21:52:55.292388 ip-10-0-140-113 kubenswrapper[2563]: I0915 21:52:55.292366    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-kj6pj_99a17472-2d2b-45ac-98c1-cce356627d5c/kube-storage-version-migrator-operator/0.log"