Sep 15 18:50:41.035906 ip-10-0-140-183 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 18:50:41.459824 ip-10-0-140-183 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:41.459824 ip-10-0-140-183 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 18:50:41.459824 ip-10-0-140-183 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:41.459824 ip-10-0-140-183 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 18:50:41.459824 ip-10-0-140-183 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:41.462234 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.462153    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 18:50:41.467240 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467224    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:41.467240 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467239    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467243    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467246    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467249    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467252    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467254    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467257    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467260    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467263    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467266    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467268    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467272    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467274    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467277    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467280    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467283    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467286    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467288    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467291    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:41.467305 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467294    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467296    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467299    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467302    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467304    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467307    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467310    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467314    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467319    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467322    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467325    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467327    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467330    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467333    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467335    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467338    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467341    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467344    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467347    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:41.467822 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467351    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467354    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467357    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467360    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467363    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467366    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467370    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467373    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467375    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467378    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467380    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467383    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467385    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467388    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467390    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467393    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467396    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467400    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467402    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467405    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:41.468283 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467408    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467410    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467413    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467415    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467419    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467422    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467424    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467427    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467430    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467432    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467435    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467437    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467440    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467442    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467445    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467447    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467450    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467453    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467455    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467458    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:41.468804 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467460    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467464    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467468    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467471    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467473    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467476    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467479    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467483    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467486    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467489    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467491    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467494    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467496    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467499    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467502    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467504    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467507    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467509    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467512    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:41.469292 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.467514    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:41.470034 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470022    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:41.470034 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470034    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470037    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470041    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470044    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470048    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470051    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470053    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470056    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470059    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470062    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470064    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470067    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470070    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470073    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470075    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470078    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470080    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470083    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470086    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470088    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:41.470091 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470091    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470093    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470097    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470100    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470102    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470105    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470108    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470110    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470113    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470116    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470118    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470121    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470123    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470126    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470128    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470132    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470134    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470137    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470139    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470141    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:41.470628 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470144    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470146    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470149    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470153    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470157    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470160    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470162    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470165    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470167    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470170    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470172    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470175    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470178    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470181    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470184    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470187    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470190    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470193    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470195    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:41.471142 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470198    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470200    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470203    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470206    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470208    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470211    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470214    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470218    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470221    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470224    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470227    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470230    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470232    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470235    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470237    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470240    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470242    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470245    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470251    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470253    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:41.471633 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470256    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470259    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470262    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470264    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470267    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470269    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470272    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470275    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470278    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470280    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470284    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470286    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470290    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470294    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470297    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470300    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470303    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470305    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470307    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470668    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:41.472137 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470674    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470676    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470679    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470682    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470685    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470687    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470690    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470693    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470695    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470698    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470701    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470704    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470706    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470709    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470712    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470715    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470717    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470720    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470723    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470726    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:41.472632 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470729    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470731    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470734    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470736    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470739    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470741    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470744    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470746    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470749    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470751    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470754    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470756    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470758    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470761    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470763    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470766    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470769    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470771    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470774    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470776    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:41.473147 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470779    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470781    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470784    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470786    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470789    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470791    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470794    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470797    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470799    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470802    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470805    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470807    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470810    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470813    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470816    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470818    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470821    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470823    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470826    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470829    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:41.473667 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470832    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470836    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470840    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470844    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470847    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470851    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470854    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470857    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470859    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470862    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470864    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470867    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470869    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470874    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470877    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470879    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470884    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470887    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:41.474154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470890    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470893    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470896    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470898    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470901    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470904    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470907    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470910    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470912    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470915    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470917    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470920    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470922    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470925    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470927    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470930    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470932    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470935    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470937    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.470940    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471019    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 18:50:41.474606 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471059    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471066    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471070    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471074    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471077    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471081    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471086    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471090    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471093    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471096    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471100    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471103    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471107    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471110    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471113    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471116    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471119    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471124    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471126    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471129    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471133    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471136    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471140    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 18:50:41.475129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471143    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471146    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471149    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471153    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471156    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471159    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471162    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471165    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471169    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471172    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471174    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471177    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471180    2556 flags.go:64] FLAG: --enable-server="true"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471183    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471187    2556 flags.go:64] FLAG: --event-burst="100"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471191    2556 flags.go:64] FLAG: --event-qps="50"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471196    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471199    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471202    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471205    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471209    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471212    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471215    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471218    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471221    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 18:50:41.475698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471224    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471227    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471230    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471233    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471235    2556 flags.go:64] FLAG: --feature-gates=""
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471239    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471242    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471245    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471249    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471252    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471255    2556 flags.go:64] FLAG: --help="false"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471257    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471261    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471264    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471267    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471270    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471273    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471276    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471279    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471281    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471284    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471288    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471291    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471295    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 15 18:50:41.476445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471298    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471301    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471304    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471307    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471310    2556 flags.go:64] FLAG: --lock-file=""
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471313    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471316    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471319    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471324    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471327    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471330    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471333    2556 flags.go:64] FLAG: --logging-format="text"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471336    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471339    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471342    2556 flags.go:64] FLAG: --manifest-url=""
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471345    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471349    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471352    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471369    2556 flags.go:64] FLAG: --max-pods="110"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471374    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471377    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471380    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471383    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471386    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471389    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 18:50:41.477068 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471392    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471399    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471402    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471405    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471408    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471411    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471416    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471420    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471424    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471427    2556 flags.go:64] FLAG: --port="10250"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471430    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471433    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c976ff8a5394bf35"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471438    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471441    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471444    2556 flags.go:64] FLAG: --register-node="true"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471447    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471451    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471453    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471456    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471459    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471462    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471465    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471468    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471471    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471474    2556 flags.go:64] FLAG: --runonce="false"
Sep 15 18:50:41.477759 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471477    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471480    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471483    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471486    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471488    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471492    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471495    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471498    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471500    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471503    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471506    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471509    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471512    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471515    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471519    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471524    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471527    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471530    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471535    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471538    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471541    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471544    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471547    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471551    2556 flags.go:64] FLAG: --v="2"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471555    2556 flags.go:64] FLAG: --version="false"
Sep 15 18:50:41.478385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471559    2556 flags.go:64] FLAG: --vmodule=""
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471562    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.471566    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471708    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471713    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471717    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471719    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471722    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471725    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471728    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471730    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471733    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471735    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471738    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471742    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471746    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471748    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471751    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471755    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:41.479395 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471758    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471761    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471764    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471767    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471769    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471773    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471775    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471778    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471780    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471784    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471786    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471789    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471792    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471794    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471797    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471800    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471802    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471805    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471807    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471810    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:41.480211 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471812    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471815    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471817    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471820    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471822    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471825    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471829    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471832    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471835    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471838    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471841    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471845    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471848    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471851    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471855    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471858    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471861    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471864    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471866    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:41.481063 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471869    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471872    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471874    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471877    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471880    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471883    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471885    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471888    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471891    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471893    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471896    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471898    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471901    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471904    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471906    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471909    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471911    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471914    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471916    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471919    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:41.481922 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471922    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471925    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471927    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471930    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471934    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471938    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471942    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471947    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471951    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471962    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471965    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471968    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471970    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471974    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471977    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471980    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471983    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471985    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471988    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:41.482764 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471991    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:41.483252 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471993    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:41.483252 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471996    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:41.483252 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.471999    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:41.483252 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.472001    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:41.483252 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.472702    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:41.483252 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.482654    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 18:50:41.483252 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.482676    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 18:50:41.483564 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.483551    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 18:50:41.483618 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.483569    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 18:50:41.483680 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483671    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:41.483680 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483680    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483686    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483691    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483695    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483701    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483704    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483709    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483714    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483719    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483723    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483728    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483732    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483736    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483740    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483745    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483750    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483754    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483758    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483763    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483767    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:41.483785 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483772    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483793    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483799    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483803    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483807    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483814    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483820    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483825    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483830    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483834    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483840    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483844    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483849    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483854    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483859    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483863    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483868    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483872    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483877    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:41.484692 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483882    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483886    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483890    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483895    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483899    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483904    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483908    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483912    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483917    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483921    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483926    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483930    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483934    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483938    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483943    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483949    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483954    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483958    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483962    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:41.485332 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483969    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483974    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483979    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483985    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483990    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483994    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.483999    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484003    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484008    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484012    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484016    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484020    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484024    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484029    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484033    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484037    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484042    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484048    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484054    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484059    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:41.486086 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484065    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484069    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484074    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484078    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484083    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484087    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484092    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484096    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484102    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484106    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484110    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484114    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484119    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484124    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484129    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484133    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484137    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484142    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484146    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:41.486681 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484151    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.484159    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484331    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484340    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484345    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484350    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484354    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484358    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484363    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484366    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484371    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484376    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484381    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484386    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484391    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:41.487198 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484396    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484400    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484404    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484409    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484413    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484418    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484423    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484428    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484433    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484437    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484442    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484447    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484453    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484459    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484465    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484471    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484475    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484480    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484485    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:41.487593 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484489    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484493    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484497    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484501    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484507    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484514    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484519    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484524    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484528    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484532    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484536    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484541    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484545    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484552    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484557    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484561    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484566    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484570    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484574    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484578    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:41.488200 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484583    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484587    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484591    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484596    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484600    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484604    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484609    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484613    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484618    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484622    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484626    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484630    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484634    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484660    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484665    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484669    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484673    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484678    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484683    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484687    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:41.489045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484691    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484695    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484699    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484704    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484708    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484712    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484718    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484722    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484726    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484730    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484738    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484743    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484749    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484754    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484759    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484763    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484769    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484773    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484778    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:41.489546 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484782    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484786    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484790    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484795    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484799    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484803    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484807    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:41.484811    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.484820    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.484967    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 18:50:41.490045 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.489757    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 18:50:41.490838 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.490826    2556 server.go:1019] "Starting client certificate rotation"
Sep 15 18:50:41.490946 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.490927    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:41.490976 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.490961    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:41.493618 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.493606    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:41.497451 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.497432    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:41.506568 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.506553    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 15 18:50:41.511008 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.510993    2556 log.go:25] "Validated CRI v1 image API"
Sep 15 18:50:41.511872 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.511849    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 18:50:41.515236 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.515203    2556 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a02f589c-07cb-4336-8b59-aaa75bfa7ef7:/dev/nvme0n1p3 e5845243-2a1c-4d81-980a-2b48f174cbe9:/dev/nvme0n1p4]
Sep 15 18:50:41.515290 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.515238    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 18:50:41.521263 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.521243    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:41.522306 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.522200    2556 manager.go:216] Machine: {Timestamp:2026-09-15 18:50:41.520526982 +0000 UTC m=+0.376380243 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094396 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2266d8cdbca82882f7f1647bc94e17 SystemUUID:ec2266d8-cdbc-a828-82f7-f1647bc94e17 BootID:f54b13b6-6032-4883-b5ab-581c0034651a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:79:df:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:79:df:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:6d:d6:2d:4a:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 18:50:41.522306 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.522301    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 18:50:41.522417 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.522405    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 18:50:41.525777 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.525755    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 18:50:41.525903 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.525780    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 18:50:41.526590 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.526581    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 18:50:41.526623 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.526593    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 18:50:41.526623 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.526600    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:41.527291 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.527282    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:41.528512 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.528503    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 18:50:41.533202 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533190    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:41.533308 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533298    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 18:50:41.533342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533329    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 18:50:41.533369 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533342    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 18:50:41.533369 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533360    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 18:50:41.533431 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533372    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 15 18:50:41.533431 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533385    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 18:50:41.533818 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533791    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:41.533818 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.533810    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:41.536105 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.536084    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 18:50:41.539007 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.538991    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 18:50:41.539096 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.539021    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 18:50:41.541151 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541139    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 18:50:41.541211 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541157    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 18:50:41.541211 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541163    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 18:50:41.541211 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541169    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 18:50:41.541211 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541177    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 18:50:41.541211 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541182    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 18:50:41.541211 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541196    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 18:50:41.541465 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541224    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 18:50:41.541465 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541232    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 18:50:41.541465 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541238    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 18:50:41.541465 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541247    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 18:50:41.541465 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.541253    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 18:50:41.542144 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.542132    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 18:50:41.542144 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.542142    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 18:50:41.544081 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.544065    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 18:50:41.544757 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.544736    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 18:50:41.544802 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.544736    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 18:50:41.546049 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.546037    2556 server.go:1268] "Started kubelet"
Sep 15 18:50:41.546131 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.546105    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 18:50:41.546272 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.546223    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 18:50:41.546402 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.546381    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 18:50:41.546953 ip-10-0-140-183 systemd[1]: Started Kubernetes Kubelet.
Sep 15 18:50:41.547728 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.547701    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 18:50:41.548763 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.548703    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 18:50:41.551412 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.551393    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 18:50:41.555326 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.555309    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:41.556029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.556013    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 18:50:41.556168 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.556116    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:41.556808 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.556792    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 18:50:41.556808 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.556808    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 18:50:41.556927 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.556795    2556 factory.go:55] Registering systemd factory
Sep 15 18:50:41.556927 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.556795    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 18:50:41.556927 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.556853    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 15 18:50:41.556927 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.556901    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 18:50:41.556927 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.556912    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 18:50:41.557081 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.557037    2556 factory.go:153] Registering CRI-O factory
Sep 15 18:50:41.557081 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.557048    2556 factory.go:223] Registration of the crio container factory successfully
Sep 15 18:50:41.557135 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.557087    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 18:50:41.557135 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.557100    2556 factory.go:103] Registering Raw factory
Sep 15 18:50:41.557135 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.557111    2556 manager.go:1196] Started watching for new ooms in manager
Sep 15 18:50:41.557439 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.557427    2556 manager.go:349] Starting recovery of all containers
Sep 15 18:50:41.560691 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.560636    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 18:50:41.560691 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.560670    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 18:50:41.561279 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.560064    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-183.ec2.internal.18d5932b65892f46  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-183.ec2.internal,UID:ip-10-0-140-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-183.ec2.internal,},FirstTimestamp:2026-09-15 18:50:41.546006342 +0000 UTC m=+0.401859601,LastTimestamp:2026-09-15 18:50:41.546006342 +0000 UTC m=+0.401859601,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-183.ec2.internal,}"
Sep 15 18:50:41.564322 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.564278    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ksgdk"
Sep 15 18:50:41.571304 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.571186    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ksgdk"
Sep 15 18:50:41.571304 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.571220    2556 manager.go:354] Recovery completed
Sep 15 18:50:41.575486 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.575469    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:41.577250 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.577236    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:41.577321 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.577266    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:41.577321 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.577278    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:41.577830 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.577813    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 18:50:41.577830 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.577829    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 18:50:41.577949 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.577833    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:41.577949 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.577845    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:41.578466 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.578453    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:41.578526 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.578477    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:41.578526 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.578490    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:41.579358 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.579285    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-183.ec2.internal.18d5932b6765fc8a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-183.ec2.internal,UID:ip-10-0-140-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-183.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-183.ec2.internal,},FirstTimestamp:2026-09-15 18:50:41.577254026 +0000 UTC m=+0.433107285,LastTimestamp:2026-09-15 18:50:41.577254026 +0000 UTC m=+0.433107285,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-183.ec2.internal,}"
Sep 15 18:50:41.579870 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.579857    2556 policy_none.go:49] "None policy: Start"
Sep 15 18:50:41.579941 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.579873    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 18:50:41.579941 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.579882    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 18:50:41.581872 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.581859    2556 policy_none.go:47] "Start"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.616971    2556 manager.go:337] "Starting Device Plugin manager"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.617007    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.617016    2556 server.go:87] "Starting device plugin registration server"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.617233    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.617259    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.617322    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.617407    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.617424    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.618086    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 18:50:41.628997 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.618134    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:41.669826 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.669798    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 18:50:41.671150 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.671131    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 18:50:41.671230 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.671181    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 18:50:41.671230 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.671205    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 18:50:41.671315 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.671281    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 18:50:41.673793 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.673770    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:41.717743 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.717700    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:41.718529 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.718516    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:41.718596 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.718542    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:41.718596 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.718551    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:41.718596 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.718570    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.727535 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.727520    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.727575 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.727541    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-183.ec2.internal\": node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:41.751146 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.751125    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:41.772283 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.772265    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal"]
Sep 15 18:50:41.772334 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.772323    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:41.773541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.773525    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:41.773595 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.773553    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:41.773595 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.773564    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:41.776092 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.776079    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:41.776211 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.776194    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.776268 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.776234    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:41.776837 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.776821    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:41.776904 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.776846    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:41.776904 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.776855    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:41.777075 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.777061    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:41.777107 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.777086    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:41.777107 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.777096    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:41.779525 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.779505    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.779610 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.779530    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:41.780158 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.780141    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:41.780242 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.780165    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:41.780242 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.780184    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:41.810303 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.810282    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-183.ec2.internal\" not found" node="ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.814598 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.814582    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-183.ec2.internal\" not found" node="ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.851935 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.851920    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:41.858804 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.858786    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2948c7237ca7a7061402e7afd2e99d8e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal\" (UID: \"2948c7237ca7a7061402e7afd2e99d8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.858863 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.858809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78a56846a26782d79f6f1b96091c0b40-config\") pod \"kube-apiserver-proxy-ip-10-0-140-183.ec2.internal\" (UID: \"78a56846a26782d79f6f1b96091c0b40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.858863 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.858827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2948c7237ca7a7061402e7afd2e99d8e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal\" (UID: \"2948c7237ca7a7061402e7afd2e99d8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.952937 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:41.952915    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:41.959239 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.959220    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2948c7237ca7a7061402e7afd2e99d8e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal\" (UID: \"2948c7237ca7a7061402e7afd2e99d8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.959318 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.959254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78a56846a26782d79f6f1b96091c0b40-config\") pod \"kube-apiserver-proxy-ip-10-0-140-183.ec2.internal\" (UID: \"78a56846a26782d79f6f1b96091c0b40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.959318 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.959279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2948c7237ca7a7061402e7afd2e99d8e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal\" (UID: \"2948c7237ca7a7061402e7afd2e99d8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.959318 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.959308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2948c7237ca7a7061402e7afd2e99d8e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal\" (UID: \"2948c7237ca7a7061402e7afd2e99d8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.959432 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.959320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78a56846a26782d79f6f1b96091c0b40-config\") pod \"kube-apiserver-proxy-ip-10-0-140-183.ec2.internal\" (UID: \"78a56846a26782d79f6f1b96091c0b40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:41.959432 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:41.959327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2948c7237ca7a7061402e7afd2e99d8e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal\" (UID: \"2948c7237ca7a7061402e7afd2e99d8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:42.053699 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.053626    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.113138 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.113113    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:42.116781 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.116761    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:42.154601 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.154446    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.255032 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.254988    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.356371 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.356285    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.463945 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.463904    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.492223 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.492196    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 18:50:42.492373 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.492355    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:42.564524 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.564487    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.577625 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.577577    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 18:45:42 +0000 UTC" deadline="2028-06-22 11:21:51.34440609 +0000 UTC"
Sep 15 18:50:42.577625 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.577621    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15496h31m8.766788454s"
Sep 15 18:50:42.665623 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.665597    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.737176 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.737152    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:42.763013 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.762987    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:42.765987 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.765970    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.769248 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:42.769215    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2948c7237ca7a7061402e7afd2e99d8e.slice/crio-cbf83ec42a7a7de4e27909754fbc389d9a42f0e46b569185f800ad5e775437e5 WatchSource:0}: Error finding container cbf83ec42a7a7de4e27909754fbc389d9a42f0e46b569185f800ad5e775437e5: Status 404 returned error can't find the container with id cbf83ec42a7a7de4e27909754fbc389d9a42f0e46b569185f800ad5e775437e5
Sep 15 18:50:42.769917 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:42.769891    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a56846a26782d79f6f1b96091c0b40.slice/crio-8070029b684f65c4cec99b03f4dcda47ff861870f7521e3641e6bbe033728f1f WatchSource:0}: Error finding container 8070029b684f65c4cec99b03f4dcda47ff861870f7521e3641e6bbe033728f1f: Status 404 returned error can't find the container with id 8070029b684f65c4cec99b03f4dcda47ff861870f7521e3641e6bbe033728f1f
Sep 15 18:50:42.775899 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:42.775877    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:50:42.866152 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.866122    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:42.966939 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:42.966878    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-183.ec2.internal\" not found"
Sep 15 18:50:43.021348 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.021325    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:43.057100 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.057072    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:43.067413 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.067390    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:50:43.068391 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.068377    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal"
Sep 15 18:50:43.079987 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.079968    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:50:43.521010 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.520921    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:43.535124 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.535100    2556 apiserver.go:52] "Watching apiserver"
Sep 15 18:50:43.543156 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.543133    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 18:50:43.543597 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.543574    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5crfs","kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bjfzd","openshift-dns/node-resolver-64626","openshift-image-registry/node-ca-sx76p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal","openshift-multus/multus-additional-cni-plugins-g7mxn","openshift-multus/network-metrics-daemon-hfjb8","kube-system/konnectivity-agent-xnqp7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb","openshift-multus/multus-hmgm4","openshift-network-diagnostics/network-check-target-k5r8j","openshift-network-operator/iptables-alerter-t6tjq","openshift-ovn-kubernetes/ovnkube-node-vfh72"]
Sep 15 18:50:43.546278 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.546251    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.548509 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.548488    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.549004 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.548982    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 18:50:43.549004 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.548990    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 18:50:43.549164 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.549080    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 18:50:43.549346 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.549329    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 18:50:43.549421 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.549409    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 18:50:43.549481 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.549424    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t26b7\""
Sep 15 18:50:43.550787 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.550765    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 18:50:43.550884 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.550807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.550952 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.550942    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:50:43.551297 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.551266    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8jbfc\""
Sep 15 18:50:43.551382 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.551333    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 18:50:43.552961 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.552939    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 18:50:43.553062 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.552947    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:50:43.553062 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.553020    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5rn97\""
Sep 15 18:50:43.553298 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.553279    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.555430 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.555407    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 18:50:43.555430 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.555414    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 18:50:43.555600 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.555502    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlffw\""
Sep 15 18:50:43.555891 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.555717    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.556271 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.556252    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:43.557716 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.557685    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 18:50:43.558200 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.558080    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8tmqk\""
Sep 15 18:50:43.558200 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.558101    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 18:50:43.558200 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.558088    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 18:50:43.558489 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.558471    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.560580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.560558    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-btcwk\""
Sep 15 18:50:43.560691 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.560558    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 18:50:43.560691 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.560681    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 18:50:43.560864 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.560820    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:50:43.562448 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.562080    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 18:50:43.562448 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.562162    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 18:50:43.562448 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.562289    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 18:50:43.562693 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.562548    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 18:50:43.564476 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.563332    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2m2lq\""
Sep 15 18:50:43.564476 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.563353    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 18:50:43.565463 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.565099    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.565463 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.565135    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 18:50:43.565463 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.565206    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:43.567733 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.567707    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:43.568080 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.568059    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:43.568169 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.568139    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:50:43.569961 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.569943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09057b3f-4b9b-4839-a4c4-f37bced3b634-hosts-file\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.570055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.569968    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.570055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.569987    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.570055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.570055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570024    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysctl-d\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.570055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysctl-conf\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570082    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-run\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570146    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-sys\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570180    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e674786-ff00-4acc-940c-90966a5fdf73-etc-tuned\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09057b3f-4b9b-4839-a4c4-f37bced3b634-tmp-dir\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570269    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-slash\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570296    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-etc-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570334    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhjs\" (UniqueName: \"kubernetes.io/projected/09057b3f-4b9b-4839-a4c4-f37bced3b634-kube-api-access-gwhjs\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-systemd-units\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.570383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570384    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovnkube-config\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570408    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovn-node-metrics-cert\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570449    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovnkube-script-lib\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570476    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-system-cni-dir\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-iptables-alerter-script\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570510    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-host-slash\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570548    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-modprobe-d\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570569    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e674786-ff00-4acc-940c-90966a5fdf73-tmp\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570585    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb454\" (UniqueName: \"kubernetes.io/projected/4e674786-ff00-4acc-940c-90966a5fdf73-kube-api-access-cb454\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570599    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-log-socket\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570619    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbcx2\" (UniqueName: \"kubernetes.io/projected/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-kube-api-access-lbcx2\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570646    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.570775 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570696    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570688    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4t99\" (UniqueName: \"kubernetes.io/projected/4f43b24a-2748-44e9-848c-10b56daaf12b-kube-api-access-b4t99\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-run-netns\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570971    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-node-log\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.570995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-cnibin\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571020    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-serviceca\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-kubelet\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571049    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-ovn\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571065    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvjk\" (UniqueName: \"kubernetes.io/projected/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-kube-api-access-zwvjk\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571091    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-lib-modules\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571111    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-var-lib-kubelet\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.571152 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-cni-bin\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571194    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-cni-netd\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571259    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-kubernetes\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571295    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-host\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-host\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5bn\" (UniqueName: \"kubernetes.io/projected/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-kube-api-access-wl5bn\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-systemd\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571376    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-var-lib-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571391    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571413    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-env-overrides\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysconfig\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.571654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.571498    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-systemd\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.573009 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.572979    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 18:50:43.573104 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.573080    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 18:50:43.573172 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.573127    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9qv9r\""
Sep 15 18:50:43.573172 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.573160    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 18:50:43.573829 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.573808    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.576312 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.576286    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:43.576410 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.576342    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:43.576914 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.576746    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 18:50:43.576914 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.576774    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rklgq\""
Sep 15 18:50:43.600574 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.600544    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-htn6w"
Sep 15 18:50:43.608379 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.608006    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-htn6w"
Sep 15 18:50:43.658496 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.658357    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 18:50:43.671630 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/790ee8dd-f692-426e-be62-4eebe5bc293c-kubelet-config\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.671764 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671637    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-sys-fs\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.671764 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671667    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-run-netns\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.671764 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671684    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-cnibin\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.671764 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671757    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-system-cni-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671783    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-socket-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671801    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-run-netns\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671811    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-kubelet\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-cnibin\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-kubelet\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671908    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-lib-modules\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671967    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.671986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-cni-netd\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672013    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672021    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672041    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-kubernetes\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672068    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-cni-netd\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5bn\" (UniqueName: \"kubernetes.io/projected/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-kube-api-access-wl5bn\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-lib-modules\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672121    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672142    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-kubernetes\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-cni-multus\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-multus-certs\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672258    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-systemd\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-etc-kubernetes\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672320    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx7fs\" (UniqueName: \"kubernetes.io/projected/b2d1e003-7ce3-4274-a974-58acb446076d-kube-api-access-sx7fs\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672379    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-systemd\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672418    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g755p\" (UniqueName: \"kubernetes.io/projected/0724acd2-fe1b-4134-817d-a4fc7300ce5c-kube-api-access-g755p\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672446    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-registration-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672470    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66b4cdae-b75b-4519-98d7-2dcb9b70fbf6-konnectivity-ca\") pod \"konnectivity-agent-xnqp7\" (UID: \"66b4cdae-b75b-4519-98d7-2dcb9b70fbf6\") " pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:50:43.672580 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672497    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysctl-d\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-sys\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672546    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e674786-ff00-4acc-940c-90966a5fdf73-etc-tuned\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672571    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-slash\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672597    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-etc-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672623    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-iptables-alerter-script\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672625    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-sys\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672653    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysctl-d\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672701    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66b4cdae-b75b-4519-98d7-2dcb9b70fbf6-agent-certs\") pod \"konnectivity-agent-xnqp7\" (UID: \"66b4cdae-b75b-4519-98d7-2dcb9b70fbf6\") " pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672713    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672737    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-etc-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhjs\" (UniqueName: \"kubernetes.io/projected/09057b3f-4b9b-4839-a4c4-f37bced3b634-kube-api-access-gwhjs\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672781    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovn-node-metrics-cert\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672811    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-host-slash\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672841    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-socket-dir-parent\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672872    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-kubelet\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.673541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-modprobe-d\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-log-socket\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672970    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbcx2\" (UniqueName: \"kubernetes.io/projected/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-kube-api-access-lbcx2\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672997    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4t99\" (UniqueName: \"kubernetes.io/projected/4f43b24a-2748-44e9-848c-10b56daaf12b-kube-api-access-b4t99\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673026    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d1e003-7ce3-4274-a974-58acb446076d-cni-binary-copy\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-netns\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673082    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-cni-bin\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-node-log\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673124    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-cnibin\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673169    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-hostroot\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673195    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673224    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-iptables-alerter-script\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673223    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjnr\" (UniqueName: \"kubernetes.io/projected/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-kube-api-access-dxjnr\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673283    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-serviceca\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673309    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-ovn\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673332    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvjk\" (UniqueName: \"kubernetes.io/projected/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-kube-api-access-zwvjk\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.674497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673350    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-modprobe-d\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673404    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-log-socket\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673482    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-conf-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-device-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673538    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-var-lib-kubelet\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-cni-bin\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673597    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-k8s-cni-cncf-io\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673620    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d1e003-7ce3-4274-a974-58acb446076d-multus-daemon-config\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673651    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-host\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673707    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-host\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.673952    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-node-log\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674026    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-host\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.672784    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-slash\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674055    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-systemd\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674087    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-var-lib-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.676454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-host-slash\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-env-overrides\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-ovn\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674183    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-var-lib-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674185    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674204    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-openvswitch\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-serviceca\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674262    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysconfig\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09057b3f-4b9b-4839-a4c4-f37bced3b634-hosts-file\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/790ee8dd-f692-426e-be62-4eebe5bc293c-dbus\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674363    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-var-lib-kubelet\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-host-cni-bin\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysctl-conf\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-run\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674486    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09057b3f-4b9b-4839-a4c4-f37bced3b634-tmp-dir\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.677350 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674503    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09057b3f-4b9b-4839-a4c4-f37bced3b634-hosts-file\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovnkube-script-lib\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674295    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-cni-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-systemd-units\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovnkube-config\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674649    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-system-cni-dir\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674673    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e674786-ff00-4acc-940c-90966a5fdf73-tmp\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674697    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb454\" (UniqueName: \"kubernetes.io/projected/4e674786-ff00-4acc-940c-90966a5fdf73-kube-api-access-cb454\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674743    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674781    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-systemd-units\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.674454    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysconfig\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675107    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09057b3f-4b9b-4839-a4c4-f37bced3b634-tmp-dir\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675245    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-env-overrides\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675316    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-run\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675362    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f43b24a-2748-44e9-848c-10b56daaf12b-system-cni-dir\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.678180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675400    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-host\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.679055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovnkube-script-lib\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.679055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675561    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-run-systemd\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.679055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675619    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e674786-ff00-4acc-940c-90966a5fdf73-etc-sysctl-conf\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.679055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.675866    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f43b24a-2748-44e9-848c-10b56daaf12b-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.679055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.676070    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovnkube-config\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.679055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.677616    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e674786-ff00-4acc-940c-90966a5fdf73-tmp\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.679055 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.678011    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e674786-ff00-4acc-940c-90966a5fdf73-etc-tuned\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.679385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.679048    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-ovn-node-metrics-cert\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.682689 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.682361    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal" event={"ID":"2948c7237ca7a7061402e7afd2e99d8e","Type":"ContainerStarted","Data":"cbf83ec42a7a7de4e27909754fbc389d9a42f0e46b569185f800ad5e775437e5"}
Sep 15 18:50:43.684099 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.684069    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5bn\" (UniqueName: \"kubernetes.io/projected/275b46c7-18c0-4a76-8af3-1b6f17b2fbec-kube-api-access-wl5bn\") pod \"node-ca-sx76p\" (UID: \"275b46c7-18c0-4a76-8af3-1b6f17b2fbec\") " pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.685287 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.685215    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbcx2\" (UniqueName: \"kubernetes.io/projected/175a2bd9-d6b5-4f02-a5b5-650a0fa125f2-kube-api-access-lbcx2\") pod \"ovnkube-node-vfh72\" (UID: \"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.685287 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.685251    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal" event={"ID":"78a56846a26782d79f6f1b96091c0b40","Type":"ContainerStarted","Data":"8070029b684f65c4cec99b03f4dcda47ff861870f7521e3641e6bbe033728f1f"}
Sep 15 18:50:43.686342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.686317    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb454\" (UniqueName: \"kubernetes.io/projected/4e674786-ff00-4acc-940c-90966a5fdf73-kube-api-access-cb454\") pod \"tuned-bjfzd\" (UID: \"4e674786-ff00-4acc-940c-90966a5fdf73\") " pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.686445 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.686319    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvjk\" (UniqueName: \"kubernetes.io/projected/b00e6c37-7c00-46ac-9b43-ff33b0e74bf1-kube-api-access-zwvjk\") pod \"iptables-alerter-t6tjq\" (UID: \"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1\") " pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.686915 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.686734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4t99\" (UniqueName: \"kubernetes.io/projected/4f43b24a-2748-44e9-848c-10b56daaf12b-kube-api-access-b4t99\") pod \"multus-additional-cni-plugins-g7mxn\" (UID: \"4f43b24a-2748-44e9-848c-10b56daaf12b\") " pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.687764 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.687739    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhjs\" (UniqueName: \"kubernetes.io/projected/09057b3f-4b9b-4839-a4c4-f37bced3b634-kube-api-access-gwhjs\") pod \"node-resolver-64626\" (UID: \"09057b3f-4b9b-4839-a4c4-f37bced3b634\") " pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.775549 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/790ee8dd-f692-426e-be62-4eebe5bc293c-kubelet-config\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.775549 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-sys-fs\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.775549 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775538    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-system-cni-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-socket-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775626    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-cni-multus\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775641    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/790ee8dd-f692-426e-be62-4eebe5bc293c-kubelet-config\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775696    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-multus-certs\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775707    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-system-cni-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775721    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-cni-multus\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775726    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-etc-kubernetes\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775768    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-multus-certs\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775779    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-sys-fs\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775767    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-socket-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775803    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx7fs\" (UniqueName: \"kubernetes.io/projected/b2d1e003-7ce3-4274-a974-58acb446076d-kube-api-access-sx7fs\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g755p\" (UniqueName: \"kubernetes.io/projected/0724acd2-fe1b-4134-817d-a4fc7300ce5c-kube-api-access-g755p\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775780    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-etc-kubernetes\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775862    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-registration-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775888    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66b4cdae-b75b-4519-98d7-2dcb9b70fbf6-konnectivity-ca\") pod \"konnectivity-agent-xnqp7\" (UID: \"66b4cdae-b75b-4519-98d7-2dcb9b70fbf6\") " pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:50:43.776227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.775959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-registration-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776035    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66b4cdae-b75b-4519-98d7-2dcb9b70fbf6-agent-certs\") pod \"konnectivity-agent-xnqp7\" (UID: \"66b4cdae-b75b-4519-98d7-2dcb9b70fbf6\") " pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776065    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-socket-dir-parent\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.776082    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-socket-dir-parent\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.776172    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:44.276133385 +0000 UTC m=+3.131986637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776197    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-kubelet\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776230    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d1e003-7ce3-4274-a974-58acb446076d-cni-binary-copy\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-netns\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776274    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-cni-bin\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776293    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-cnibin\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776314    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-hostroot\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776334    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776359    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjnr\" (UniqueName: \"kubernetes.io/projected/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-kube-api-access-dxjnr\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776382    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-cni-bin\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776389    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.777163 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776422    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-hostroot\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-conf-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.776456    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776456    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66b4cdae-b75b-4519-98d7-2dcb9b70fbf6-konnectivity-ca\") pod \"konnectivity-agent-xnqp7\" (UID: \"66b4cdae-b75b-4519-98d7-2dcb9b70fbf6\") " pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-conf-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776490    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-cnibin\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.776523    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret podName:790ee8dd-f692-426e-be62-4eebe5bc293c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:44.276508243 +0000 UTC m=+3.132361516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret") pod "global-pull-secret-syncer-5crfs" (UID: "790ee8dd-f692-426e-be62-4eebe5bc293c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-netns\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776532    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-var-lib-kubelet\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776559    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-device-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-k8s-cni-cncf-io\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776613    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d1e003-7ce3-4274-a974-58acb446076d-multus-daemon-config\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776680    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-host-run-k8s-cni-cncf-io\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-device-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776715    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/790ee8dd-f692-426e-be62-4eebe5bc293c-dbus\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776749    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.778077 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776771    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/790ee8dd-f692-426e-be62-4eebe5bc293c-dbus\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:43.778945 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776778    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-cni-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778945 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776820    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.778945 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776860    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d1e003-7ce3-4274-a974-58acb446076d-multus-cni-dir\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778945 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.776903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d1e003-7ce3-4274-a974-58acb446076d-cni-binary-copy\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.778945 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.777216    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d1e003-7ce3-4274-a974-58acb446076d-multus-daemon-config\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.779244 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.779225    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66b4cdae-b75b-4519-98d7-2dcb9b70fbf6-agent-certs\") pod \"konnectivity-agent-xnqp7\" (UID: \"66b4cdae-b75b-4519-98d7-2dcb9b70fbf6\") " pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:50:43.782624 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.782603    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:43.782742 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.782628    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:43.782742 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.782643    2556 projected.go:196] Error preparing data for projected volume kube-api-access-cqs4j for pod openshift-network-diagnostics/network-check-target-k5r8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:43.782742 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:43.782727    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j podName:9029f176-9478-4faa-9fb5-233822b89b22 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:44.282709104 +0000 UTC m=+3.138562379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cqs4j" (UniqueName: "kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j") pod "network-check-target-k5r8j" (UID: "9029f176-9478-4faa-9fb5-233822b89b22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:43.785827 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.785716    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g755p\" (UniqueName: \"kubernetes.io/projected/0724acd2-fe1b-4134-817d-a4fc7300ce5c-kube-api-access-g755p\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:43.786218 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.786191    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx7fs\" (UniqueName: \"kubernetes.io/projected/b2d1e003-7ce3-4274-a974-58acb446076d-kube-api-access-sx7fs\") pod \"multus-hmgm4\" (UID: \"b2d1e003-7ce3-4274-a974-58acb446076d\") " pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:43.786309 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.786199    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjnr\" (UniqueName: \"kubernetes.io/projected/abf8c3fd-47df-4e09-a147-aefa7b5d2fd8-kube-api-access-dxjnr\") pod \"aws-ebs-csi-driver-node-42mhb\" (UID: \"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.859492 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.859456    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g7mxn"
Sep 15 18:50:43.870020 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.869754    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t6tjq"
Sep 15 18:50:43.878335 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.878313    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjfzd"
Sep 15 18:50:43.884718 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.884692    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-64626"
Sep 15 18:50:43.892643 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.892621    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sx76p"
Sep 15 18:50:43.904050 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.903890    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:50:43.912881 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.912853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:50:43.918799 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.918779    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb"
Sep 15 18:50:43.924783 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:43.924759    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hmgm4"
Sep 15 18:50:44.139790 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.139756    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09057b3f_4b9b_4839_a4c4_f37bced3b634.slice/crio-22ebfc4d9ccb50c0b795724b534cfebe45096f25444fb4d016d50b479cc5e3e7 WatchSource:0}: Error finding container 22ebfc4d9ccb50c0b795724b534cfebe45096f25444fb4d016d50b479cc5e3e7: Status 404 returned error can't find the container with id 22ebfc4d9ccb50c0b795724b534cfebe45096f25444fb4d016d50b479cc5e3e7
Sep 15 18:50:44.141254 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.141208    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf8c3fd_47df_4e09_a147_aefa7b5d2fd8.slice/crio-1e525f0bd75cad9030c8e4e9d86d7763fd31fb6d28837221615f32e6442d88a2 WatchSource:0}: Error finding container 1e525f0bd75cad9030c8e4e9d86d7763fd31fb6d28837221615f32e6442d88a2: Status 404 returned error can't find the container with id 1e525f0bd75cad9030c8e4e9d86d7763fd31fb6d28837221615f32e6442d88a2
Sep 15 18:50:44.145967 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.144476    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275b46c7_18c0_4a76_8af3_1b6f17b2fbec.slice/crio-3f4becd7f814b44d036b2e2b01619accb7a63b691c5a3c8d5d45fb851ab23696 WatchSource:0}: Error finding container 3f4becd7f814b44d036b2e2b01619accb7a63b691c5a3c8d5d45fb851ab23696: Status 404 returned error can't find the container with id 3f4becd7f814b44d036b2e2b01619accb7a63b691c5a3c8d5d45fb851ab23696
Sep 15 18:50:44.148475 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.148445    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00e6c37_7c00_46ac_9b43_ff33b0e74bf1.slice/crio-8649ffb3a508acfbf0fb8d5a10c3043fdafa8ea2762c9de3a78457e6d0cdf3b8 WatchSource:0}: Error finding container 8649ffb3a508acfbf0fb8d5a10c3043fdafa8ea2762c9de3a78457e6d0cdf3b8: Status 404 returned error can't find the container with id 8649ffb3a508acfbf0fb8d5a10c3043fdafa8ea2762c9de3a78457e6d0cdf3b8
Sep 15 18:50:44.149312 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.149287    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175a2bd9_d6b5_4f02_a5b5_650a0fa125f2.slice/crio-966767b9f7abeb5e0e728ccf2faf39954be005cacd4e9f148cb14ac432f5835b WatchSource:0}: Error finding container 966767b9f7abeb5e0e728ccf2faf39954be005cacd4e9f148cb14ac432f5835b: Status 404 returned error can't find the container with id 966767b9f7abeb5e0e728ccf2faf39954be005cacd4e9f148cb14ac432f5835b
Sep 15 18:50:44.150557 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.150523    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f43b24a_2748_44e9_848c_10b56daaf12b.slice/crio-1c8970e663e0d5cc33d5e18accbbf61a406b6cd992df0845f4567d50db141804 WatchSource:0}: Error finding container 1c8970e663e0d5cc33d5e18accbbf61a406b6cd992df0845f4567d50db141804: Status 404 returned error can't find the container with id 1c8970e663e0d5cc33d5e18accbbf61a406b6cd992df0845f4567d50db141804
Sep 15 18:50:44.174430 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.174409    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e674786_ff00_4acc_940c_90966a5fdf73.slice/crio-b5e64de0e73fb6050957bc5c369abf583209bfed7e11ce4a107c748ebb2d6705 WatchSource:0}: Error finding container b5e64de0e73fb6050957bc5c369abf583209bfed7e11ce4a107c748ebb2d6705: Status 404 returned error can't find the container with id b5e64de0e73fb6050957bc5c369abf583209bfed7e11ce4a107c748ebb2d6705
Sep 15 18:50:44.175194 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.175112    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d1e003_7ce3_4274_a974_58acb446076d.slice/crio-24a43f3944664cb841fa0c001d28da5f45ab369fc5c7ac85f0ee426c92cf51bf WatchSource:0}: Error finding container 24a43f3944664cb841fa0c001d28da5f45ab369fc5c7ac85f0ee426c92cf51bf: Status 404 returned error can't find the container with id 24a43f3944664cb841fa0c001d28da5f45ab369fc5c7ac85f0ee426c92cf51bf
Sep 15 18:50:44.175917 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:50:44.175896    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b4cdae_b75b_4519_98d7_2dcb9b70fbf6.slice/crio-4781f4cfa2a8f45852464785dcc6ddedc40a7cb1688beb1fe9885cfdc799a103 WatchSource:0}: Error finding container 4781f4cfa2a8f45852464785dcc6ddedc40a7cb1688beb1fe9885cfdc799a103: Status 404 returned error can't find the container with id 4781f4cfa2a8f45852464785dcc6ddedc40a7cb1688beb1fe9885cfdc799a103
Sep 15 18:50:44.281284 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.281248    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:44.281460 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.281354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:44.281460 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:44.281394    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:44.281562 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:44.281469    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret podName:790ee8dd-f692-426e-be62-4eebe5bc293c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:45.281453331 +0000 UTC m=+4.137306584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret") pod "global-pull-secret-syncer-5crfs" (UID: "790ee8dd-f692-426e-be62-4eebe5bc293c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:44.281562 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:44.281473    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:44.281562 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:44.281525    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:45.281509235 +0000 UTC m=+4.137362492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:44.382742 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.382709    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:44.382928 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:44.382876    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:44.382928 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:44.382905    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:44.382928 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:44.382916    2556 projected.go:196] Error preparing data for projected volume kube-api-access-cqs4j for pod openshift-network-diagnostics/network-check-target-k5r8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:44.383078 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:44.382971    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j podName:9029f176-9478-4faa-9fb5-233822b89b22 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:45.3829558 +0000 UTC m=+4.238809053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqs4j" (UniqueName: "kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j") pod "network-check-target-k5r8j" (UID: "9029f176-9478-4faa-9fb5-233822b89b22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:44.612175 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.612028    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:45:44 +0000 UTC" deadline="2028-06-04 02:25:09.831809764 +0000 UTC"
Sep 15 18:50:44.612175 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.612073    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15055h34m25.219741788s"
Sep 15 18:50:44.688915 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.688865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-64626" event={"ID":"09057b3f-4b9b-4839-a4c4-f37bced3b634","Type":"ContainerStarted","Data":"22ebfc4d9ccb50c0b795724b534cfebe45096f25444fb4d016d50b479cc5e3e7"}
Sep 15 18:50:44.690293 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.690260    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnqp7" event={"ID":"66b4cdae-b75b-4519-98d7-2dcb9b70fbf6","Type":"ContainerStarted","Data":"4781f4cfa2a8f45852464785dcc6ddedc40a7cb1688beb1fe9885cfdc799a103"}
Sep 15 18:50:44.691659 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.691627    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"966767b9f7abeb5e0e728ccf2faf39954be005cacd4e9f148cb14ac432f5835b"}
Sep 15 18:50:44.692995 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.692967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjfzd" event={"ID":"4e674786-ff00-4acc-940c-90966a5fdf73","Type":"ContainerStarted","Data":"b5e64de0e73fb6050957bc5c369abf583209bfed7e11ce4a107c748ebb2d6705"}
Sep 15 18:50:44.694314 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.694288    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" event={"ID":"4f43b24a-2748-44e9-848c-10b56daaf12b","Type":"ContainerStarted","Data":"1c8970e663e0d5cc33d5e18accbbf61a406b6cd992df0845f4567d50db141804"}
Sep 15 18:50:44.695635 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.695602    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sx76p" event={"ID":"275b46c7-18c0-4a76-8af3-1b6f17b2fbec","Type":"ContainerStarted","Data":"3f4becd7f814b44d036b2e2b01619accb7a63b691c5a3c8d5d45fb851ab23696"}
Sep 15 18:50:44.696870 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.696848    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb" event={"ID":"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8","Type":"ContainerStarted","Data":"1e525f0bd75cad9030c8e4e9d86d7763fd31fb6d28837221615f32e6442d88a2"}
Sep 15 18:50:44.698180 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.698158    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hmgm4" event={"ID":"b2d1e003-7ce3-4274-a974-58acb446076d","Type":"ContainerStarted","Data":"24a43f3944664cb841fa0c001d28da5f45ab369fc5c7ac85f0ee426c92cf51bf"}
Sep 15 18:50:44.699452 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:44.699428    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t6tjq" event={"ID":"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1","Type":"ContainerStarted","Data":"8649ffb3a508acfbf0fb8d5a10c3043fdafa8ea2762c9de3a78457e6d0cdf3b8"}
Sep 15 18:50:45.291014 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.290699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:45.291014 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.290965    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:45.291014 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.291003    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:45.291294 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.291048    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:47.291026085 +0000 UTC m=+6.146879351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:45.291294 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.291107    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:45.291294 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.291165    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret podName:790ee8dd-f692-426e-be62-4eebe5bc293c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:47.291149836 +0000 UTC m=+6.147003102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret") pod "global-pull-secret-syncer-5crfs" (UID: "790ee8dd-f692-426e-be62-4eebe5bc293c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:45.392396 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.392365    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:45.392543 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.392519    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:45.392543 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.392540    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:45.392669 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.392555    2556 projected.go:196] Error preparing data for projected volume kube-api-access-cqs4j for pod openshift-network-diagnostics/network-check-target-k5r8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:45.392669 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.392611    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j podName:9029f176-9478-4faa-9fb5-233822b89b22 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:47.392591271 +0000 UTC m=+6.248444521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqs4j" (UniqueName: "kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j") pod "network-check-target-k5r8j" (UID: "9029f176-9478-4faa-9fb5-233822b89b22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:45.612949 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.612834    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:45:44 +0000 UTC" deadline="2028-05-09 14:49:15.508343542 +0000 UTC"
Sep 15 18:50:45.612949 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.612897    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14443h58m29.895450234s"
Sep 15 18:50:45.674535 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.673875    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:45.674535 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.674031    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:45.674535 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.674217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:45.674535 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.674217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:45.674535 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.674350    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:50:45.674535 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:45.674418    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:45.730505 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.729802    2556 generic.go:356] "Generic (PLEG): container finished" podID="2948c7237ca7a7061402e7afd2e99d8e" containerID="6026a17605d85d592c5f8883531f715e79d79c9a25a4e015705bbae066f34085" exitCode=0
Sep 15 18:50:45.730505 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.730232    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal" event={"ID":"2948c7237ca7a7061402e7afd2e99d8e","Type":"ContainerDied","Data":"6026a17605d85d592c5f8883531f715e79d79c9a25a4e015705bbae066f34085"}
Sep 15 18:50:45.739168 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:45.739142    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal" event={"ID":"78a56846a26782d79f6f1b96091c0b40","Type":"ContainerStarted","Data":"f3b3da5b518eef9fb9e9825dbb3e08bd7156cc0ba1a16a65f8251f5e10a8108d"}
Sep 15 18:50:46.757746 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:46.757642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal" event={"ID":"2948c7237ca7a7061402e7afd2e99d8e","Type":"ContainerStarted","Data":"089e6d7d4ca5675dc80750edae87d4605de6949d59e9c72858190651c949b5ca"}
Sep 15 18:50:46.772992 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:46.772938    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-183.ec2.internal" podStartSLOduration=3.7729175379999997 podStartE2EDuration="3.772917538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:45.764140718 +0000 UTC m=+4.619993990" watchObservedRunningTime="2026-09-15 18:50:46.772917538 +0000 UTC m=+5.628770810"
Sep 15 18:50:46.773927 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:46.773300    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-183.ec2.internal" podStartSLOduration=3.773289809 podStartE2EDuration="3.773289809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:46.772385362 +0000 UTC m=+5.628238635" watchObservedRunningTime="2026-09-15 18:50:46.773289809 +0000 UTC m=+5.629143080"
Sep 15 18:50:47.309038 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:47.308998    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:47.309312 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:47.309064    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:47.309312 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.309247    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:47.309464 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.309315    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret podName:790ee8dd-f692-426e-be62-4eebe5bc293c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:51.309296552 +0000 UTC m=+10.165149806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret") pod "global-pull-secret-syncer-5crfs" (UID: "790ee8dd-f692-426e-be62-4eebe5bc293c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:47.309464 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.309430    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:47.309586 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.309470    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:51.309456324 +0000 UTC m=+10.165309571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:47.409968 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:47.409900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:47.410198 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.410167    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:47.410198 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.410193    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:47.410318 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.410207    2556 projected.go:196] Error preparing data for projected volume kube-api-access-cqs4j for pod openshift-network-diagnostics/network-check-target-k5r8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:47.410318 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.410273    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j podName:9029f176-9478-4faa-9fb5-233822b89b22 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:51.410253417 +0000 UTC m=+10.266106665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqs4j" (UniqueName: "kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j") pod "network-check-target-k5r8j" (UID: "9029f176-9478-4faa-9fb5-233822b89b22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:47.671794 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:47.671710    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:47.671948 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.671854    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:47.672184 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:47.672154    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:47.672329 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.672288    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:47.672385 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:47.672352    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:47.672462 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:47.672446    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:50:49.672346 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:49.672293    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:49.672862 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:49.672432    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:49.672862 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:49.672298    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:49.672862 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:49.672532    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:49.672862 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:49.672293    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:49.672862 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:49.672634    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:50:51.341548 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:51.341514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:51.342126 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:51.341564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:51.342126 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.341679    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:51.342126 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.341747    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:59.3417272 +0000 UTC m=+18.197580450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:51.342126 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.341687    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:51.342126 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.341794    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret podName:790ee8dd-f692-426e-be62-4eebe5bc293c nodeName:}" failed. No retries permitted until 2026-09-15 18:50:59.341784246 +0000 UTC m=+18.197637497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret") pod "global-pull-secret-syncer-5crfs" (UID: "790ee8dd-f692-426e-be62-4eebe5bc293c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:51.442203 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:51.442164    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:51.442380 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.442328    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:51.442380 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.442349    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:51.442380 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.442360    2556 projected.go:196] Error preparing data for projected volume kube-api-access-cqs4j for pod openshift-network-diagnostics/network-check-target-k5r8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:51.442532 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.442417    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j podName:9029f176-9478-4faa-9fb5-233822b89b22 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:59.442398092 +0000 UTC m=+18.298251363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqs4j" (UniqueName: "kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j") pod "network-check-target-k5r8j" (UID: "9029f176-9478-4faa-9fb5-233822b89b22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:51.672722 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:51.672620    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:51.672890 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.672807    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:50:51.673256 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:51.673229    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:51.673386 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.673326    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:51.673463 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:51.673437    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:51.673539 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:51.673517    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:53.672197 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:53.671702    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:53.672197 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:53.671728    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:53.672197 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:53.671701    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:53.672197 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:53.671845    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:50:53.672197 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:53.671944    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:53.672197 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:53.672072    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:55.671933 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:55.671897    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:55.672386 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:55.671897    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:55.672386 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:55.672037    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:55.672386 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:55.672117    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:55.672386 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:55.672159    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:55.672386 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:55.672278    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:50:57.672451 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:57.672392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:57.672936 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:57.672392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:57.672936 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:57.672529    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:57.672936 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:57.672533    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:57.672936 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:57.672654    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:50:57.672936 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:57.672711    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:59.402823 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:59.402779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:59.403361 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:59.402842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:59.403361 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.402896    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:59.403361 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.402951    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:59.403361 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.402967    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:51:15.40294752 +0000 UTC m=+34.258800769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:59.403361 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.402985    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret podName:790ee8dd-f692-426e-be62-4eebe5bc293c nodeName:}" failed. No retries permitted until 2026-09-15 18:51:15.4029754 +0000 UTC m=+34.258828650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret") pod "global-pull-secret-syncer-5crfs" (UID: "790ee8dd-f692-426e-be62-4eebe5bc293c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:59.504060 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:59.504026    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:59.504217 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.504153    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:59.504217 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.504168    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:59.504217 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.504179    2556 projected.go:196] Error preparing data for projected volume kube-api-access-cqs4j for pod openshift-network-diagnostics/network-check-target-k5r8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:59.504346 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.504229    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j podName:9029f176-9478-4faa-9fb5-233822b89b22 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:15.504216604 +0000 UTC m=+34.360069850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqs4j" (UniqueName: "kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j") pod "network-check-target-k5r8j" (UID: "9029f176-9478-4faa-9fb5-233822b89b22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:59.671691 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:59.671597    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:50:59.671691 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:59.671636    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:50:59.671881 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:50:59.671597    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:50:59.671881 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.671747    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:50:59.671881 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.671851    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:50:59.672040 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:50:59.671948    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:01.673514 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:01.673479    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:01.673953 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:01.673572    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:01.673953 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:01.673611    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:01.673953 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:01.673636    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:01.673953 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:01.673701    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:51:01.673953 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:01.673776    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:51:02.791918 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:02.791169    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnqp7" event={"ID":"66b4cdae-b75b-4519-98d7-2dcb9b70fbf6","Type":"ContainerStarted","Data":"6a6a3d55c5fe0192694a448cd38dd000d2247b9b6313e02f17eac7bcaa037a3d"}
Sep 15 18:51:02.794216 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:02.793968    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sx76p" event={"ID":"275b46c7-18c0-4a76-8af3-1b6f17b2fbec","Type":"ContainerStarted","Data":"968d78475cac5a09eeb72d8aa9f96eb07a6e6e63b9e65dc08e9502f57084ea58"}
Sep 15 18:51:02.798436 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:02.797996    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb" event={"ID":"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8","Type":"ContainerStarted","Data":"2c819910644ffc4ccbcf97432704ea92a85322d6d7970a69ebf0a1ecfcd04d0e"}
Sep 15 18:51:02.808264 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:02.807926    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xnqp7" podStartSLOduration=2.458316137 podStartE2EDuration="20.807907673s" totalImagesPullingTime="18.349591536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:02.807007856 +0000 UTC m=+21.662861124" watchObservedRunningTime="2026-09-15 18:51:02.807907673 +0000 UTC m=+21.663760942"
Sep 15 18:51:02.822558 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:02.822506    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sx76p" podStartSLOduration=2.389327361 podStartE2EDuration="20.82249064s" totalImagesPullingTime="18.433163279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:02.822030889 +0000 UTC m=+21.677884157" watchObservedRunningTime="2026-09-15 18:51:02.82249064 +0000 UTC m=+21.678343939"
Sep 15 18:51:03.672534 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.672369    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:03.672751 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.672369    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:03.672751 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:03.672629    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:03.672751 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.672436    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:03.672915 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:03.672724    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:51:03.672915 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:03.672824    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:51:03.804715 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.804685    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 18:51:03.805160 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.805131    2556 generic.go:356] "Generic (PLEG): container finished" podID="175a2bd9-d6b5-4f02-a5b5-650a0fa125f2" containerID="aed1379193fc6ff3174f75513f5982b6352b0b1537d07c46a121d30faaf5c49e" exitCode=1
Sep 15 18:51:03.805305 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.805208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"5e5f3b8f4870094da852d158834713b0abc4ec819345988f80ba08e9f57a674b"}
Sep 15 18:51:03.807909 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.805686    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"6090c7b8a9b83b683b27479b9032a45453e4eadaca4169dafbc7406bbad723f7"}
Sep 15 18:51:03.807909 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.805777    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"a9cc050a61f330ebd05d3f1c448cb1ae8d23c78f355d46a893c1eb50641eea13"}
Sep 15 18:51:03.807909 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.805915    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"f20860269d080024ccc45c7a64601e47772278d974cd2ea0095234e311eab26b"}
Sep 15 18:51:03.807909 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.806003    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerDied","Data":"aed1379193fc6ff3174f75513f5982b6352b0b1537d07c46a121d30faaf5c49e"}
Sep 15 18:51:03.807909 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.806067    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"8d0e403d0e340914c09b39742fe0ed64f970515be8c5e0a134de167ea65ffec9"}
Sep 15 18:51:03.809510 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.809479    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjfzd" event={"ID":"4e674786-ff00-4acc-940c-90966a5fdf73","Type":"ContainerStarted","Data":"46f60181c5cc4448a2800ee346bc9d8762e53be18e16815a5b6a49ec0a5fae46"}
Sep 15 18:51:03.811394 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.811367    2556 generic.go:356] "Generic (PLEG): container finished" podID="4f43b24a-2748-44e9-848c-10b56daaf12b" containerID="fcb2f5911dd4804c28fdbd01dc02dae76f9b2a2348b3b77301ccc520942ea152" exitCode=0
Sep 15 18:51:03.811492 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.811450    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" event={"ID":"4f43b24a-2748-44e9-848c-10b56daaf12b","Type":"ContainerDied","Data":"fcb2f5911dd4804c28fdbd01dc02dae76f9b2a2348b3b77301ccc520942ea152"}
Sep 15 18:51:03.813618 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.813596    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hmgm4" event={"ID":"b2d1e003-7ce3-4274-a974-58acb446076d","Type":"ContainerStarted","Data":"16fe88eaea26644e3000641bbc71dd148449dccb9c63fc34cab579b8f092388e"}
Sep 15 18:51:03.815297 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.815275    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-64626" event={"ID":"09057b3f-4b9b-4839-a4c4-f37bced3b634","Type":"ContainerStarted","Data":"042593e7469669c90315d3619f07609b7839553c8eab6d9594f66e7b6b51215a"}
Sep 15 18:51:03.824909 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.824873    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bjfzd" podStartSLOduration=3.420127301 podStartE2EDuration="21.824861652s" totalImagesPullingTime="18.404734351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:03.824833253 +0000 UTC m=+22.680686521" watchObservedRunningTime="2026-09-15 18:51:03.824861652 +0000 UTC m=+22.680714899"
Sep 15 18:51:03.862894 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.862819    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hmgm4" podStartSLOduration=2.854432321 podStartE2EDuration="21.86280627s" totalImagesPullingTime="19.008373949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:03.839449851 +0000 UTC m=+22.695303119" watchObservedRunningTime="2026-09-15 18:51:03.86280627 +0000 UTC m=+22.718659570"
Sep 15 18:51:03.879103 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:03.879063    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-64626" podStartSLOduration=3.435032789 podStartE2EDuration="21.879050816s" totalImagesPullingTime="18.444018027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:03.878796866 +0000 UTC m=+22.734650136" watchObservedRunningTime="2026-09-15 18:51:03.879050816 +0000 UTC m=+22.734904085"
Sep 15 18:51:04.403343 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.403317    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:51:04.403888 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.403870    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:51:04.407837 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.407820    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 18:51:04.629403 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.629277    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T18:51:04.407834481Z","UUID":"a0a64a10-e971-4aee-b8b0-8c2b19f343d9","Handler":null,"Name":"","Endpoint":""}
Sep 15 18:51:04.630810 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.630792    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 18:51:04.630935 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.630821    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 18:51:04.818535 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.818502    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t6tjq" event={"ID":"b00e6c37-7c00-46ac-9b43-ff33b0e74bf1","Type":"ContainerStarted","Data":"9ab5b02c867703b8e2b58b944bc887ec7c8f0b29234402686878dd90ada4550a"}
Sep 15 18:51:04.820471 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.820448    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb" event={"ID":"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8","Type":"ContainerStarted","Data":"85351fd69ac382ed5c831f1e0854a225ea62223fe502c59de0b74a63bb97b676"}
Sep 15 18:51:04.820965 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.820924    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:51:04.821092 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.821077    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xnqp7"
Sep 15 18:51:04.846348 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:04.846307    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t6tjq" podStartSLOduration=4.436559288 podStartE2EDuration="22.846291326s" totalImagesPullingTime="18.409732038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:04.833310069 +0000 UTC m=+23.689163336" watchObservedRunningTime="2026-09-15 18:51:04.846291326 +0000 UTC m=+23.702144666"
Sep 15 18:51:05.672160 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:05.672134    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:05.672260 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:05.672138    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:05.672317 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:05.672261    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:51:05.672375 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:05.672352    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:05.672433 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:05.672135    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:05.672482 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:05.672442    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:51:05.827698 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:05.827661    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb" event={"ID":"abf8c3fd-47df-4e09-a147-aefa7b5d2fd8","Type":"ContainerStarted","Data":"d810b8b450ce61fe267e2b072436400a874192673d1e8d127c5a572ebff84ff1"}
Sep 15 18:51:05.832413 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:05.832387    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 18:51:05.832876 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:05.832843    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"2bcd081c8d015e92adf701e264c9cb71797664a5aec8d336163a12c62cc837ed"}
Sep 15 18:51:05.844493 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:05.844447    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42mhb" podStartSLOduration=2.7171933839999998 podStartE2EDuration="23.844433694s" totalImagesPullingTime="21.12724031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:05.843880537 +0000 UTC m=+24.699733808" watchObservedRunningTime="2026-09-15 18:51:05.844433694 +0000 UTC m=+24.700286998"
Sep 15 18:51:07.671951 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:07.671913    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:07.672397 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:07.671913    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:07.672397 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:07.672054    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:51:07.672397 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:07.671925    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:07.672397 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:07.672109    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:51:07.672397 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:07.672200    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:09.672243 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.672181    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:09.672668 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.672181    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:09.672668 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:09.672301    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:51:09.672668 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.672322    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:09.672668 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:09.672376    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:09.672668 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:09.672421    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:51:09.844848 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.844821    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 18:51:09.845221 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.845199    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"d644dbedba3c1baedb5b0d5f943fd356244e5a10e26f5773b516f8d52b3481ce"}
Sep 15 18:51:09.845429 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.845412    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:51:09.845661 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.845611    2556 scope.go:117] "RemoveContainer" containerID="aed1379193fc6ff3174f75513f5982b6352b0b1537d07c46a121d30faaf5c49e"
Sep 15 18:51:09.847237 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.847218    2556 generic.go:356] "Generic (PLEG): container finished" podID="4f43b24a-2748-44e9-848c-10b56daaf12b" containerID="1f5dffdd6ff47a24811b82281bc538dda63ea865b66ed872009f7213afb46f40" exitCode=0
Sep 15 18:51:09.847343 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.847256    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" event={"ID":"4f43b24a-2748-44e9-848c-10b56daaf12b","Type":"ContainerDied","Data":"1f5dffdd6ff47a24811b82281bc538dda63ea865b66ed872009f7213afb46f40"}
Sep 15 18:51:09.861516 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:09.861496    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:51:10.855607 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:10.855442    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 18:51:10.856253 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:10.856129    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" event={"ID":"175a2bd9-d6b5-4f02-a5b5-650a0fa125f2","Type":"ContainerStarted","Data":"580b1e6d72218bf9699b6d9a8cd29f63fa65738c7177614eb8aa86f8baf183b6"}
Sep 15 18:51:10.856517 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:10.856489    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:51:10.856655 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:10.856588    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:51:10.874052 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:10.874022    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:51:10.890729 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:10.890675    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72" podStartSLOduration=10.470271016 podStartE2EDuration="28.89063562s" totalImagesPullingTime="18.420364604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:10.890139879 +0000 UTC m=+29.745993146" watchObservedRunningTime="2026-09-15 18:51:10.89063562 +0000 UTC m=+29.746488888"
Sep 15 18:51:11.032216 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:11.032186    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hfjb8"]
Sep 15 18:51:11.032382 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:11.032311    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:11.032451 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:11.032427    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:11.035322 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:11.035295    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k5r8j"]
Sep 15 18:51:11.035462 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:11.035423    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:11.035551 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:11.035530    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:51:11.035877 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:11.035850    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5crfs"]
Sep 15 18:51:11.035999 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:11.035950    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:11.036132 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:11.036026    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:51:11.860535 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:11.860451    2556 generic.go:356] "Generic (PLEG): container finished" podID="4f43b24a-2748-44e9-848c-10b56daaf12b" containerID="338132d1905209f5de445806d66b6478b3d285878fae470b20fcc7fa2a6f9e8c" exitCode=0
Sep 15 18:51:11.861167 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:11.860532    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" event={"ID":"4f43b24a-2748-44e9-848c-10b56daaf12b","Type":"ContainerDied","Data":"338132d1905209f5de445806d66b6478b3d285878fae470b20fcc7fa2a6f9e8c"}
Sep 15 18:51:12.672208 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:12.672133    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:12.672360 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:12.672132    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:12.672360 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:12.672132    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:12.672360 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:12.672340    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:51:12.672507 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:12.672253    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:51:12.672507 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:12.672407    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:13.867450 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:13.867282    2556 generic.go:356] "Generic (PLEG): container finished" podID="4f43b24a-2748-44e9-848c-10b56daaf12b" containerID="7b5a80bc7248457b0002805b0096faef1e307a70faf134d31d25d80387a18ac3" exitCode=0
Sep 15 18:51:13.867881 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:13.867354    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" event={"ID":"4f43b24a-2748-44e9-848c-10b56daaf12b","Type":"ContainerDied","Data":"7b5a80bc7248457b0002805b0096faef1e307a70faf134d31d25d80387a18ac3"}
Sep 15 18:51:14.671587 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:14.671554    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:14.671768 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:14.671686    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfs" podUID="790ee8dd-f692-426e-be62-4eebe5bc293c"
Sep 15 18:51:14.671850 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:14.671764    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:14.671911 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:14.671889    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:51:14.671969 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:14.671941    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:14.672034 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:14.672016    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5r8j" podUID="9029f176-9478-4faa-9fb5-233822b89b22"
Sep 15 18:51:15.420602 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:15.420571    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:15.421034 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:15.420657    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:15.421034 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:15.420733    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:15.421034 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:15.420769    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:15.421034 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:15.420796    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:51:47.420781652 +0000 UTC m=+66.276634897 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:15.421034 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:15.420817    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret podName:790ee8dd-f692-426e-be62-4eebe5bc293c nodeName:}" failed. No retries permitted until 2026-09-15 18:51:47.420803328 +0000 UTC m=+66.276656574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret") pod "global-pull-secret-syncer-5crfs" (UID: "790ee8dd-f692-426e-be62-4eebe5bc293c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:15.521094 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:15.521064    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:15.521280 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:15.521241    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:15.521280 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:15.521262    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:15.521280 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:15.521274    2556 projected.go:196] Error preparing data for projected volume kube-api-access-cqs4j for pod openshift-network-diagnostics/network-check-target-k5r8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:15.521419 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:15.521327    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j podName:9029f176-9478-4faa-9fb5-233822b89b22 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:47.521307387 +0000 UTC m=+66.377160647 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqs4j" (UniqueName: "kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j") pod "network-check-target-k5r8j" (UID: "9029f176-9478-4faa-9fb5-233822b89b22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:15.992306 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:15.992228    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-183.ec2.internal" event="NodeReady"
Sep 15 18:51:15.992451 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:15.992379    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 18:51:16.024005 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.023976    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76c555597d-46l4z"]
Sep 15 18:51:16.043388 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.043346    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c555597d-46l4z"]
Sep 15 18:51:16.043537 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.043429    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"]
Sep 15 18:51:16.043683 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.043658    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.046183 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.046161    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 18:51:16.046566 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.046399    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwrs6\""
Sep 15 18:51:16.046566 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.046449    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 18:51:16.046566 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.046453    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 18:51:16.052374 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.052349    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 18:51:16.062607 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.062587    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jf5l9"]
Sep 15 18:51:16.062782 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.062764    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:16.065252 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.065190    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 18:51:16.065252 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.065207    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 18:51:16.065404 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.065266    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rr57c\""
Sep 15 18:51:16.075002 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.074983    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n5k94"]
Sep 15 18:51:16.075110 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.075092    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.077765 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.077722    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 18:51:16.077855 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.077773    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hj774\""
Sep 15 18:51:16.077855 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.077774    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 18:51:16.091489 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.091463    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"]
Sep 15 18:51:16.091577 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.091496    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jf5l9"]
Sep 15 18:51:16.091577 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.091516    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n5k94"]
Sep 15 18:51:16.091667 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.091599    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n5k94"
Sep 15 18:51:16.094024 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.094005    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 18:51:16.094122 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.094041    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xmcwz\""
Sep 15 18:51:16.094122 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.094051    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 18:51:16.094122 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.094075    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 18:51:16.127027 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.126997    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-installation-pull-secrets\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.127165 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.127055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-image-registry-private-configuration\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.127165 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.127116    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.127165 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.127155    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-bound-sa-token\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.127287 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.127212    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-trusted-ca\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.127287 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.127251    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-certificates\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.127287 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.127276    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzxt\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-kube-api-access-vtzxt\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.127408 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.127316    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-ca-trust-extracted\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.228478 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228434    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-ca-trust-extracted\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.228673 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/300c7133-fd1a-4a29-95da-f0ebaaf366da-cert\") pod \"ingress-canary-n5k94\" (UID: \"300c7133-fd1a-4a29-95da-f0ebaaf366da\") " pod="openshift-ingress-canary/ingress-canary-n5k94"
Sep 15 18:51:16.228673 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228555    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-installation-pull-secrets\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.228673 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228589    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.228673 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl49n\" (UniqueName: \"kubernetes.io/projected/300c7133-fd1a-4a29-95da-f0ebaaf366da-kube-api-access-kl49n\") pod \"ingress-canary-n5k94\" (UID: \"300c7133-fd1a-4a29-95da-f0ebaaf366da\") " pod="openshift-ingress-canary/ingress-canary-n5k94"
Sep 15 18:51:16.228673 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:16.228919 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-image-registry-private-configuration\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.228919 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228775    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.228919 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228805    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-bound-sa-token\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.228919 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228873    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-trusted-ca\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.229085 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-certificates\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.229085 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228960    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzxt\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-kube-api-access-vtzxt\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.229085 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.228992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-tmp-dir\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.229085 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.229029    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fe06272-485c-452f-8eda-13dfb438a19a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:16.229085 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.229061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-config-volume\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.229295 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.229087    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h5rt\" (UniqueName: \"kubernetes.io/projected/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-kube-api-access-9h5rt\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.231585 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.229953    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-ca-trust-extracted\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.231585 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.230476    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-certificates\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.231585 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.230587    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:16.231585 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.230607    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:51:16.231585 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.230700    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:16.730667859 +0000 UTC m=+35.586521109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:51:16.233397 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.233313    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-trusted-ca\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.234551 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.234525    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-installation-pull-secrets\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.234659 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.234527    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-image-registry-private-configuration\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.238485 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.238466    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-bound-sa-token\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.241546 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.241523    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzxt\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-kube-api-access-vtzxt\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.329489 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-tmp-dir\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.329489 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329452    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fe06272-485c-452f-8eda-13dfb438a19a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:16.329489 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329474    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-config-volume\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.329796 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h5rt\" (UniqueName: \"kubernetes.io/projected/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-kube-api-access-9h5rt\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.329796 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329541    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/300c7133-fd1a-4a29-95da-f0ebaaf366da-cert\") pod \"ingress-canary-n5k94\" (UID: \"300c7133-fd1a-4a29-95da-f0ebaaf366da\") " pod="openshift-ingress-canary/ingress-canary-n5k94"
Sep 15 18:51:16.329796 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329565    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.329796 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329587    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl49n\" (UniqueName: \"kubernetes.io/projected/300c7133-fd1a-4a29-95da-f0ebaaf366da-kube-api-access-kl49n\") pod \"ingress-canary-n5k94\" (UID: \"300c7133-fd1a-4a29-95da-f0ebaaf366da\") " pod="openshift-ingress-canary/ingress-canary-n5k94"
Sep 15 18:51:16.329796 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329615    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:16.329796 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.329786    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:51:16.330054 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.329790    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:16.330054 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.329827    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-tmp-dir\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.330054 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.329861    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:51:16.829842929 +0000 UTC m=+35.685696193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:51:16.330054 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.329880    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:16.829870468 +0000 UTC m=+35.685723722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:16.330267 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.330078    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-config-volume\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.330322 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.330294    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fe06272-485c-452f-8eda-13dfb438a19a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:16.332292 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.332268    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/300c7133-fd1a-4a29-95da-f0ebaaf366da-cert\") pod \"ingress-canary-n5k94\" (UID: \"300c7133-fd1a-4a29-95da-f0ebaaf366da\") " pod="openshift-ingress-canary/ingress-canary-n5k94"
Sep 15 18:51:16.338074 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.338052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl49n\" (UniqueName: \"kubernetes.io/projected/300c7133-fd1a-4a29-95da-f0ebaaf366da-kube-api-access-kl49n\") pod \"ingress-canary-n5k94\" (UID: \"300c7133-fd1a-4a29-95da-f0ebaaf366da\") " pod="openshift-ingress-canary/ingress-canary-n5k94"
Sep 15 18:51:16.338304 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.338279    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h5rt\" (UniqueName: \"kubernetes.io/projected/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-kube-api-access-9h5rt\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.401334 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.401304    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n5k94"
Sep 15 18:51:16.671688 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.671600    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:16.671688 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.671625    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:16.671688 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.671631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:16.674146 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.674113    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:16.674270 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.674183    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:16.674270 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.674239    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:16.674512 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.674468    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bc8c\""
Sep 15 18:51:16.674606 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.674561    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:16.674691 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.674626    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mrmrg\""
Sep 15 18:51:16.707943 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.707920    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n5k94"]
Sep 15 18:51:16.733129 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.733106    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:16.733268 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.733239    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:16.733268 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.733256    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:51:16.733368 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.733328    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:17.733313034 +0000 UTC m=+36.589166280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:51:16.834193 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.834164    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:16.834193 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:16.834198    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:16.834483 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.834342    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:51:16.834483 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.834403    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:51:17.834388624 +0000 UTC m=+36.690241892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:51:16.834483 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.834342    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:16.834666 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:16.834504    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:17.834483944 +0000 UTC m=+36.690337210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:17.744092 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:17.744053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:17.744667 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:17.744237    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:17.744667 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:17.744260    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:51:17.744667 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:17.744327    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:19.744307112 +0000 UTC m=+38.600160361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:51:17.845008 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:17.844970    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:17.845187 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:17.845021    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:17.845187 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:17.845150    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:17.845286 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:17.845205    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:51:17.845286 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:17.845232    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:19.84521018 +0000 UTC m=+38.701063436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:17.845286 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:17.845253    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:51:19.845241436 +0000 UTC m=+38.701094703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:51:18.184311 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:51:18.184062    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300c7133_fd1a_4a29_95da_f0ebaaf366da.slice/crio-4335a85010e81207d80275dab2239ff9d671486ef868c5e1bf6c43eeff2a4cd0 WatchSource:0}: Error finding container 4335a85010e81207d80275dab2239ff9d671486ef868c5e1bf6c43eeff2a4cd0: Status 404 returned error can't find the container with id 4335a85010e81207d80275dab2239ff9d671486ef868c5e1bf6c43eeff2a4cd0
Sep 15 18:51:18.881749 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:18.881678    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n5k94" event={"ID":"300c7133-fd1a-4a29-95da-f0ebaaf366da","Type":"ContainerStarted","Data":"4335a85010e81207d80275dab2239ff9d671486ef868c5e1bf6c43eeff2a4cd0"}
Sep 15 18:51:18.884624 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:18.884600    2556 generic.go:356] "Generic (PLEG): container finished" podID="4f43b24a-2748-44e9-848c-10b56daaf12b" containerID="cee69db7290cfd882de98213314ae7aa934208d5aa70707b1d0f5c1d078e2684" exitCode=0
Sep 15 18:51:18.884760 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:18.884652    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" event={"ID":"4f43b24a-2748-44e9-848c-10b56daaf12b","Type":"ContainerDied","Data":"cee69db7290cfd882de98213314ae7aa934208d5aa70707b1d0f5c1d078e2684"}
Sep 15 18:51:19.761422 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:19.761384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:19.761568 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:19.761554    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:19.761663 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:19.761570    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:51:19.761663 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:19.761631    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:23.761612645 +0000 UTC m=+42.617465911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:51:19.861933 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:19.861849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:19.861933 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:19.861899    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:19.862128 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:19.862028    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:51:19.862128 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:19.862094    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:51:23.862075472 +0000 UTC m=+42.717928721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:51:19.862128 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:19.862028    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:19.862289 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:19.862170    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:23.862149792 +0000 UTC m=+42.718003039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:19.890827 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:19.890796    2556 generic.go:356] "Generic (PLEG): container finished" podID="4f43b24a-2748-44e9-848c-10b56daaf12b" containerID="14292ba6e694237644837f3abce261dded87170347c29fd34acc45c62947a276" exitCode=0
Sep 15 18:51:19.891231 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:19.890845    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" event={"ID":"4f43b24a-2748-44e9-848c-10b56daaf12b","Type":"ContainerDied","Data":"14292ba6e694237644837f3abce261dded87170347c29fd34acc45c62947a276"}
Sep 15 18:51:20.897976 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:20.897896    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" event={"ID":"4f43b24a-2748-44e9-848c-10b56daaf12b","Type":"ContainerStarted","Data":"d6217e19604dab46d91cfa6b537fcf5183415f784ea8c97b025f60f092d21514"}
Sep 15 18:51:20.919980 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:20.919926    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g7mxn" podStartSLOduration=8.121726168 podStartE2EDuration="38.919908012s" totalImagesPullingTime="30.798181844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:20.918435911 +0000 UTC m=+39.774289181" watchObservedRunningTime="2026-09-15 18:51:20.919908012 +0000 UTC m=+39.775761279"
Sep 15 18:51:21.901914 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:21.901880    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n5k94" event={"ID":"300c7133-fd1a-4a29-95da-f0ebaaf366da","Type":"ContainerStarted","Data":"c519e5d61b5a7a3aa0a44fa019a681aa11b726acc58fdb4e1a83d5002ff803e9"}
Sep 15 18:51:21.917031 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:21.916984    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n5k94" podStartSLOduration=2.797616956 podStartE2EDuration="5.916967889s" totalImagesPullingTime="3.119350933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:21.916213895 +0000 UTC m=+40.772067166" watchObservedRunningTime="2026-09-15 18:51:21.916967889 +0000 UTC m=+40.772821262"
Sep 15 18:51:23.792490 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:23.792458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:23.792861 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:23.792579    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:23.792861 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:23.792590    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:51:23.792861 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:23.792653    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:31.792623819 +0000 UTC m=+50.648477066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:51:23.893242 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:23.893212    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:23.893412 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:23.893247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:23.893412 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:23.893358    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:23.893482 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:23.893414    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:31.893399845 +0000 UTC m=+50.749253095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:23.893482 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:23.893358    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:51:23.893552 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:23.893495    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:51:31.893482933 +0000 UTC m=+50.749336186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:51:31.842848 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:31.842808    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:31.843203 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:31.842956    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:31.843203 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:31.842977    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:51:31.843203 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:31.843033    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:47.843017498 +0000 UTC m=+66.698870744 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:51:31.943809 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:31.943782    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:31.943934 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:31.943814    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:31.943934 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:31.943918    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:51:31.943934 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:31.943921    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:31.944028 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:31.943972    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:51:47.943954381 +0000 UTC m=+66.799807630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:51:31.944028 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:31.943986    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:47.943978969 +0000 UTC m=+66.799832215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:42.884164 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:42.884130    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfh72"
Sep 15 18:51:47.454912 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.454870    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:51:47.455326 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.454933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:47.457098 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.457078    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:47.457220 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.457204    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:47.465948 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.465926    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:51:47.466060 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.466047    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:52:51.46602554 +0000 UTC m=+130.321878800 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : secret "metrics-daemon-secret" not found
Sep 15 18:51:47.468875 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.468829    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/790ee8dd-f692-426e-be62-4eebe5bc293c-original-pull-secret\") pod \"global-pull-secret-syncer-5crfs\" (UID: \"790ee8dd-f692-426e-be62-4eebe5bc293c\") " pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:47.555824 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.555792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:47.558275 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.558260    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:47.568727 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.568711    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:47.578846 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.578821    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqs4j\" (UniqueName: \"kubernetes.io/projected/9029f176-9478-4faa-9fb5-233822b89b22-kube-api-access-cqs4j\") pod \"network-check-target-k5r8j\" (UID: \"9029f176-9478-4faa-9fb5-233822b89b22\") " pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:47.585632 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.585615    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mrmrg\""
Sep 15 18:51:47.594484 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.594470    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:47.595694 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.595681    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfs"
Sep 15 18:51:47.857879 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.857813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:51:47.858011 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.857939    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:47.858011 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.857951    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:51:47.858011 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.858000    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:52:19.857986946 +0000 UTC m=+98.713840191 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:51:47.864099 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.864075    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5crfs"]
Sep 15 18:51:47.867309 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.867276    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k5r8j"]
Sep 15 18:51:47.867527 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:51:47.867507    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790ee8dd_f692_426e_be62_4eebe5bc293c.slice/crio-7d9b24df48efe631904081384db9958010d6fb329a7949800a6decf31e232547 WatchSource:0}: Error finding container 7d9b24df48efe631904081384db9958010d6fb329a7949800a6decf31e232547: Status 404 returned error can't find the container with id 7d9b24df48efe631904081384db9958010d6fb329a7949800a6decf31e232547
Sep 15 18:51:47.869846 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:51:47.869824    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9029f176_9478_4faa_9fb5_233822b89b22.slice/crio-93b47463553d7c4e1a811aaf33679fce35488cc87b7d7029147118e82e2437bd WatchSource:0}: Error finding container 93b47463553d7c4e1a811aaf33679fce35488cc87b7d7029147118e82e2437bd: Status 404 returned error can't find the container with id 93b47463553d7c4e1a811aaf33679fce35488cc87b7d7029147118e82e2437bd
Sep 15 18:51:47.958605 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.958578    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:51:47.958735 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.958610    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:51:47.958788 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.958734    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:47.958788 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.958746    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:51:47.958788 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.958787    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:52:19.958772831 +0000 UTC m=+98.814626101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:51:47.958904 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:51:47.958804    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:52:19.958794294 +0000 UTC m=+98.814647543 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:47.962290 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.962267    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5crfs" event={"ID":"790ee8dd-f692-426e-be62-4eebe5bc293c","Type":"ContainerStarted","Data":"7d9b24df48efe631904081384db9958010d6fb329a7949800a6decf31e232547"}
Sep 15 18:51:47.963478 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:47.963459    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k5r8j" event={"ID":"9029f176-9478-4faa-9fb5-233822b89b22","Type":"ContainerStarted","Data":"93b47463553d7c4e1a811aaf33679fce35488cc87b7d7029147118e82e2437bd"}
Sep 15 18:51:52.979600 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:52.979563    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5crfs" event={"ID":"790ee8dd-f692-426e-be62-4eebe5bc293c","Type":"ContainerStarted","Data":"9524658512fa1c69cd68e5d1d3e88981c0ad3fe41ecc939a0c8000738b837708"}
Sep 15 18:51:52.981189 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:52.981164    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k5r8j" event={"ID":"9029f176-9478-4faa-9fb5-233822b89b22","Type":"ContainerStarted","Data":"d181258973362cb662d4e8da15249380ee2297d6c886da5fffb54ca59e5ae2cb"}
Sep 15 18:51:52.981313 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:52.981214    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:51:52.993792 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:52.993539    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5crfs" podStartSLOduration=65.991635827 podStartE2EDuration="1m10.993520456s" totalImagesPullingTime="5.001884629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:52.993508508 +0000 UTC m=+71.849361776" watchObservedRunningTime="2026-09-15 18:51:52.993520456 +0000 UTC m=+71.849373726"
Sep 15 18:51:53.007816 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:51:53.007772    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k5r8j" podStartSLOduration=66.012852165 podStartE2EDuration="1m11.007756405s" totalImagesPullingTime="4.99490424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:53.007151998 +0000 UTC m=+71.863005266" watchObservedRunningTime="2026-09-15 18:51:53.007756405 +0000 UTC m=+71.863609700"
Sep 15 18:52:19.880440 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:52:19.880328    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:52:19.880440 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:19.880439    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:52:19.880916 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:19.880449    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:52:19.880916 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:19.880497    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:53:23.880482906 +0000 UTC m=+162.736336151 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:52:19.981331 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:52:19.981292    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:52:19.981418 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:52:19.981339    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:52:19.981458 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:19.981437    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:52:19.981458 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:19.981449    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:52:19.981517 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:19.981509    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:53:23.981491889 +0000 UTC m=+162.837345134 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:52:19.981555 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:19.981524    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:53:23.981518088 +0000 UTC m=+162.837371334 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:52:23.987029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:52:23.986995    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k5r8j"
Sep 15 18:52:51.503789 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:52:51.503747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:52:51.504246 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:51.503902    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:52:51.504246 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:52:51.503974    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:54:53.50395759 +0000 UTC m=+252.359810848 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : secret "metrics-daemon-secret" not found
Sep 15 18:53:19.055521 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:19.055478    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-76c555597d-46l4z" podUID="e66c4f4a-c7e9-4a1f-9313-721feeee1c90"
Sep 15 18:53:19.073033 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:19.073008    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl" podUID="1fe06272-485c-452f-8eda-13dfb438a19a"
Sep 15 18:53:19.084286 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:19.084267    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jf5l9" podUID="cdaba30b-50d4-422c-88d4-ac0f3dfb01c8"
Sep 15 18:53:19.188497 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:53:19.188472    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:53:19.188608 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:53:19.188473    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:53:19.689604 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:19.689568    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:53:23.922271 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:53:23.922229    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:53:23.922688 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:23.922368    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:53:23.922688 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:23.922385    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c555597d-46l4z: secret "image-registry-tls" not found
Sep 15 18:53:23.922688 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:23.922462    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls podName:e66c4f4a-c7e9-4a1f-9313-721feeee1c90 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:25.922446305 +0000 UTC m=+284.778299550 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls") pod "image-registry-76c555597d-46l4z" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90") : secret "image-registry-tls" not found
Sep 15 18:53:24.023349 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:53:24.023317    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:53:24.023349 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:53:24.023351    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:53:24.023508 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:24.023456    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 18:53:24.023508 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:24.023503    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert podName:1fe06272-485c-452f-8eda-13dfb438a19a nodeName:}" failed. No retries permitted until 2026-09-15 18:55:26.023489489 +0000 UTC m=+284.879342735 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-p4frl" (UID: "1fe06272-485c-452f-8eda-13dfb438a19a") : secret "networking-console-plugin-cert" not found
Sep 15 18:53:24.023576 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:24.023456    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:53:24.023576 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:53:24.023530    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls podName:cdaba30b-50d4-422c-88d4-ac0f3dfb01c8 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:26.023524368 +0000 UTC m=+284.879377613 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls") pod "dns-default-jf5l9" (UID: "cdaba30b-50d4-422c-88d4-ac0f3dfb01c8") : secret "dns-default-metrics-tls" not found
Sep 15 18:53:32.671550 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:53:32.671511    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:53:34.671654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:53:34.671566    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:54:52.561389 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.561358    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"]
Sep 15 18:54:52.564148 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.564126    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz"]
Sep 15 18:54:52.564278 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.564236    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.566467 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.566447    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 18:54:52.566850 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.566829    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj"]
Sep 15 18:54:52.566958 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.566935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz"
Sep 15 18:54:52.567449 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.567430    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 18:54:52.567449 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.567444    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 18:54:52.567619 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.567452    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-shz5k\""
Sep 15 18:54:52.567619 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.567454    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:54:52.569480 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.569464    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-8n8xn"]
Sep 15 18:54:52.569569 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.569556    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj"
Sep 15 18:54:52.569788 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.569768    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vvptf\""
Sep 15 18:54:52.571721 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.571702    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 15 18:54:52.571813 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.571744    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:54:52.571813 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.571766    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ghvkl\""
Sep 15 18:54:52.572088 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.572076    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.574237 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.574219    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 15 18:54:52.574333 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.574279    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h7fwd\""
Sep 15 18:54:52.574399 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.574348    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 18:54:52.574479 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.574462    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 15 18:54:52.574552 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.574534    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 18:54:52.580320 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.580300    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"]
Sep 15 18:54:52.581318 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.581297    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 15 18:54:52.581440 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.581418    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj"]
Sep 15 18:54:52.582072 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.582048    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-8n8xn"]
Sep 15 18:54:52.592362 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.592337    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz"]
Sep 15 18:54:52.661037 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.661015    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"]
Sep 15 18:54:52.663884 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.663863    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.665719 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.665701    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"]
Sep 15 18:54:52.667251 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.667229    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 18:54:52.667334 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.667259    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 15 18:54:52.667334 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.667281    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 18:54:52.667334 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.667260    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 15 18:54:52.667463 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.667232    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-92s45\""
Sep 15 18:54:52.668256 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.668239    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-g4gm9"]
Sep 15 18:54:52.668342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.668328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:52.670294 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.670242    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 15 18:54:52.670676 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.670660    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8hh7d\""
Sep 15 18:54:52.670962 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.670945    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:54:52.671035 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.670983    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.671035 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.671019    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 15 18:54:52.673200 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.673183    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 15 18:54:52.673276 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.673251    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 15 18:54:52.673353 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.673339    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 15 18:54:52.673422 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.673380    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:54:52.673492 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.673479    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gxkqm\""
Sep 15 18:54:52.676973 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.676956    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"]
Sep 15 18:54:52.678778 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.678758    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 15 18:54:52.679830 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.679735    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"]
Sep 15 18:54:52.680628 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.680610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-g4gm9"]
Sep 15 18:54:52.726374 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726350    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27cc90e0-3da1-4615-96f3-fa7b0f89488b-serving-cert\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.726464 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726375    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27cc90e0-3da1-4615-96f3-fa7b0f89488b-config\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.726464 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726398    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4dj\" (UniqueName: \"kubernetes.io/projected/f470421a-2714-4f71-9eb4-f2365287496a-kube-api-access-5v4dj\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.726464 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726451    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f470421a-2714-4f71-9eb4-f2365287496a-snapshots\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.726565 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f470421a-2714-4f71-9eb4-f2365287496a-tmp\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.726596 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726563    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f470421a-2714-4f71-9eb4-f2365287496a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.726628 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f470421a-2714-4f71-9eb4-f2365287496a-serving-cert\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.726697 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726678    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bb8z\" (UniqueName: \"kubernetes.io/projected/27cc90e0-3da1-4615-96f3-fa7b0f89488b-kube-api-access-5bb8z\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.726742 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726705    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f470421a-2714-4f71-9eb4-f2365287496a-service-ca-bundle\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.726742 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726732    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jpqd\" (UniqueName: \"kubernetes.io/projected/3d82b329-5733-4457-b3c1-2171f98e9ff8-kube-api-access-9jpqd\") pod \"volume-data-source-validator-86695b7cc7-shdhj\" (UID: \"3d82b329-5733-4457-b3c1-2171f98e9ff8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj"
Sep 15 18:54:52.726802 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.726752    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f82cr\" (UniqueName: \"kubernetes.io/projected/c3f1b537-3323-415f-9d4d-ba4b82642629-kube-api-access-f82cr\") pod \"network-check-source-6f8b768455-9qdtz\" (UID: \"c3f1b537-3323-415f-9d4d-ba4b82642629\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz"
Sep 15 18:54:52.827191 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/270f6947-813e-465b-9ad8-8428cf35b988-config\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.827191 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827172    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27cc90e0-3da1-4615-96f3-fa7b0f89488b-serving-cert\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.827191 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27cc90e0-3da1-4615-96f3-fa7b0f89488b-config\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.827321 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827212    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4dj\" (UniqueName: \"kubernetes.io/projected/f470421a-2714-4f71-9eb4-f2365287496a-kube-api-access-5v4dj\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.827321 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827276    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:52.827321 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827296    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7w9l\" (UniqueName: \"kubernetes.io/projected/adf1bcc7-1575-44ed-a4c7-273bf659fc11-kube-api-access-l7w9l\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:52.827439 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jpqd\" (UniqueName: \"kubernetes.io/projected/3d82b329-5733-4457-b3c1-2171f98e9ff8-kube-api-access-9jpqd\") pod \"volume-data-source-validator-86695b7cc7-shdhj\" (UID: \"3d82b329-5733-4457-b3c1-2171f98e9ff8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj"
Sep 15 18:54:52.827439 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827373    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rm4\" (UniqueName: \"kubernetes.io/projected/270f6947-813e-465b-9ad8-8428cf35b988-kube-api-access-52rm4\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.827439 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f82cr\" (UniqueName: \"kubernetes.io/projected/c3f1b537-3323-415f-9d4d-ba4b82642629-kube-api-access-f82cr\") pod \"network-check-source-6f8b768455-9qdtz\" (UID: \"c3f1b537-3323-415f-9d4d-ba4b82642629\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz"
Sep 15 18:54:52.827575 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/270f6947-813e-465b-9ad8-8428cf35b988-serving-cert\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.827575 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827476    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/270f6947-813e-465b-9ad8-8428cf35b988-trusted-ca\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.827575 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827511    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh2kb\" (UniqueName: \"kubernetes.io/projected/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-kube-api-access-gh2kb\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.827575 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827567    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.827795 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f470421a-2714-4f71-9eb4-f2365287496a-tmp\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.827795 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f470421a-2714-4f71-9eb4-f2365287496a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.827795 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827729    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bb8z\" (UniqueName: \"kubernetes.io/projected/27cc90e0-3da1-4615-96f3-fa7b0f89488b-kube-api-access-5bb8z\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.827795 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.827795 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827768    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27cc90e0-3da1-4615-96f3-fa7b0f89488b-config\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.828026 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827801    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f470421a-2714-4f71-9eb4-f2365287496a-serving-cert\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.828026 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827828    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f470421a-2714-4f71-9eb4-f2365287496a-service-ca-bundle\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.828026 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.827884    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f470421a-2714-4f71-9eb4-f2365287496a-snapshots\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.828159 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.828091    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f470421a-2714-4f71-9eb4-f2365287496a-tmp\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.828361 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.828337    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f470421a-2714-4f71-9eb4-f2365287496a-service-ca-bundle\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.828599 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.828574    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f470421a-2714-4f71-9eb4-f2365287496a-snapshots\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.828753 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.828585    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f470421a-2714-4f71-9eb4-f2365287496a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.829707 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.829687    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27cc90e0-3da1-4615-96f3-fa7b0f89488b-serving-cert\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.830014 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.830001    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f470421a-2714-4f71-9eb4-f2365287496a-serving-cert\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.835967 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.835941    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bb8z\" (UniqueName: \"kubernetes.io/projected/27cc90e0-3da1-4615-96f3-fa7b0f89488b-kube-api-access-5bb8z\") pod \"service-ca-operator-cf7c776b7-jw4jw\" (UID: \"27cc90e0-3da1-4615-96f3-fa7b0f89488b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.836442 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.836418    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4dj\" (UniqueName: \"kubernetes.io/projected/f470421a-2714-4f71-9eb4-f2365287496a-kube-api-access-5v4dj\") pod \"insights-operator-5844b856db-8n8xn\" (UID: \"f470421a-2714-4f71-9eb4-f2365287496a\") " pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.836813 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.836794    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f82cr\" (UniqueName: \"kubernetes.io/projected/c3f1b537-3323-415f-9d4d-ba4b82642629-kube-api-access-f82cr\") pod \"network-check-source-6f8b768455-9qdtz\" (UID: \"c3f1b537-3323-415f-9d4d-ba4b82642629\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz"
Sep 15 18:54:52.836876 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.836821    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jpqd\" (UniqueName: \"kubernetes.io/projected/3d82b329-5733-4457-b3c1-2171f98e9ff8-kube-api-access-9jpqd\") pod \"volume-data-source-validator-86695b7cc7-shdhj\" (UID: \"3d82b329-5733-4457-b3c1-2171f98e9ff8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj"
Sep 15 18:54:52.874592 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.874572    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"
Sep 15 18:54:52.881602 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.881580    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz"
Sep 15 18:54:52.888236 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.888216    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj"
Sep 15 18:54:52.898072 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.898049    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-8n8xn"
Sep 15 18:54:52.929398 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/270f6947-813e-465b-9ad8-8428cf35b988-config\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.929529 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:52.929529 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7w9l\" (UniqueName: \"kubernetes.io/projected/adf1bcc7-1575-44ed-a4c7-273bf659fc11-kube-api-access-l7w9l\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:52.929529 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rm4\" (UniqueName: \"kubernetes.io/projected/270f6947-813e-465b-9ad8-8428cf35b988-kube-api-access-52rm4\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.929529 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/270f6947-813e-465b-9ad8-8428cf35b988-serving-cert\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.929529 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/270f6947-813e-465b-9ad8-8428cf35b988-trusted-ca\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.929822 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh2kb\" (UniqueName: \"kubernetes.io/projected/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-kube-api-access-gh2kb\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.929822 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929579    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.929822 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.929663    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.929974 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:52.929830    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 18:54:52.929974 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:52.929897    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls podName:bd436ebb-24ed-4151-aa7c-c2baa894e4ff nodeName:}" failed. No retries permitted until 2026-09-15 18:54:53.429877095 +0000 UTC m=+252.285730358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-58mhq" (UID: "bd436ebb-24ed-4151-aa7c-c2baa894e4ff") : secret "cluster-monitoring-operator-tls" not found
Sep 15 18:54:52.932729 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:52.931134    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 18:54:52.932729 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.931185    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/270f6947-813e-465b-9ad8-8428cf35b988-config\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.932729 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:52.931198    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls podName:adf1bcc7-1575-44ed-a4c7-273bf659fc11 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:53.431183246 +0000 UTC m=+252.287036497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vmdj5" (UID: "adf1bcc7-1575-44ed-a4c7-273bf659fc11") : secret "samples-operator-tls" not found
Sep 15 18:54:52.932729 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.932454    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/270f6947-813e-465b-9ad8-8428cf35b988-trusted-ca\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.932729 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.932675    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.935895 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.935769    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/270f6947-813e-465b-9ad8-8428cf35b988-serving-cert\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.943705 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.943680    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7w9l\" (UniqueName: \"kubernetes.io/projected/adf1bcc7-1575-44ed-a4c7-273bf659fc11-kube-api-access-l7w9l\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:52.944371 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.944343    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rm4\" (UniqueName: \"kubernetes.io/projected/270f6947-813e-465b-9ad8-8428cf35b988-kube-api-access-52rm4\") pod \"console-operator-696884fb9c-g4gm9\" (UID: \"270f6947-813e-465b-9ad8-8428cf35b988\") " pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:52.944371 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.944367    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh2kb\" (UniqueName: \"kubernetes.io/projected/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-kube-api-access-gh2kb\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:52.988191 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:52.988165    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:54:53.146265 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.146229    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw"]
Sep 15 18:54:53.149537 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:54:53.149512    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cc90e0_3da1_4615_96f3_fa7b0f89488b.slice/crio-d34ce488c004a16aa42b8bf90fd0065e31ae2614fb24328c5737d96bdd8dc243 WatchSource:0}: Error finding container d34ce488c004a16aa42b8bf90fd0065e31ae2614fb24328c5737d96bdd8dc243: Status 404 returned error can't find the container with id d34ce488c004a16aa42b8bf90fd0065e31ae2614fb24328c5737d96bdd8dc243
Sep 15 18:54:53.154331 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.154312    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz"]
Sep 15 18:54:53.156710 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:54:53.156687    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f1b537_3323_415f_9d4d_ba4b82642629.slice/crio-633061afe75a18b1b48c723ad8d4678d94cbd59a7bd574961be66f334f1081f5 WatchSource:0}: Error finding container 633061afe75a18b1b48c723ad8d4678d94cbd59a7bd574961be66f334f1081f5: Status 404 returned error can't find the container with id 633061afe75a18b1b48c723ad8d4678d94cbd59a7bd574961be66f334f1081f5
Sep 15 18:54:53.174455 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.174434    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj"]
Sep 15 18:54:53.189549 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:54:53.189524    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d82b329_5733_4457_b3c1_2171f98e9ff8.slice/crio-3246261d26d57190c5ff4a571ecdbbf091e4c2519a500e6a4479d0f8881b50ab WatchSource:0}: Error finding container 3246261d26d57190c5ff4a571ecdbbf091e4c2519a500e6a4479d0f8881b50ab: Status 404 returned error can't find the container with id 3246261d26d57190c5ff4a571ecdbbf091e4c2519a500e6a4479d0f8881b50ab
Sep 15 18:54:53.212084 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.212061    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-8n8xn"]
Sep 15 18:54:53.213869 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:54:53.213844    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf470421a_2714_4f71_9eb4_f2365287496a.slice/crio-89eba820db450720e2501aee32e08f0226f4e1992df541e54e4a7d0775282316 WatchSource:0}: Error finding container 89eba820db450720e2501aee32e08f0226f4e1992df541e54e4a7d0775282316: Status 404 returned error can't find the container with id 89eba820db450720e2501aee32e08f0226f4e1992df541e54e4a7d0775282316
Sep 15 18:54:53.255626 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.255601    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-g4gm9"]
Sep 15 18:54:53.258255 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:54:53.258234    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270f6947_813e_465b_9ad8_8428cf35b988.slice/crio-5cd03b241b3af369e96de508702db6eedb36f8314f81f7b244c6f70ec9b7dbe0 WatchSource:0}: Error finding container 5cd03b241b3af369e96de508702db6eedb36f8314f81f7b244c6f70ec9b7dbe0: Status 404 returned error can't find the container with id 5cd03b241b3af369e96de508702db6eedb36f8314f81f7b244c6f70ec9b7dbe0
Sep 15 18:54:53.410903 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.410830    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9" event={"ID":"270f6947-813e-465b-9ad8-8428cf35b988","Type":"ContainerStarted","Data":"5cd03b241b3af369e96de508702db6eedb36f8314f81f7b244c6f70ec9b7dbe0"}
Sep 15 18:54:53.412305 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.412283    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz" event={"ID":"c3f1b537-3323-415f-9d4d-ba4b82642629","Type":"ContainerStarted","Data":"0b881f980ee02df579e9d08c03476aa89d684d1db30bb6935f37e4653c758558"}
Sep 15 18:54:53.412408 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.412319    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz" event={"ID":"c3f1b537-3323-415f-9d4d-ba4b82642629","Type":"ContainerStarted","Data":"633061afe75a18b1b48c723ad8d4678d94cbd59a7bd574961be66f334f1081f5"}
Sep 15 18:54:53.413486 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.413466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-8n8xn" event={"ID":"f470421a-2714-4f71-9eb4-f2365287496a","Type":"ContainerStarted","Data":"89eba820db450720e2501aee32e08f0226f4e1992df541e54e4a7d0775282316"}
Sep 15 18:54:53.414491 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.414472    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw" event={"ID":"27cc90e0-3da1-4615-96f3-fa7b0f89488b","Type":"ContainerStarted","Data":"d34ce488c004a16aa42b8bf90fd0065e31ae2614fb24328c5737d96bdd8dc243"}
Sep 15 18:54:53.415565 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.415538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj" event={"ID":"3d82b329-5733-4457-b3c1-2171f98e9ff8","Type":"ContainerStarted","Data":"3246261d26d57190c5ff4a571ecdbbf091e4c2519a500e6a4479d0f8881b50ab"}
Sep 15 18:54:53.424258 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.424210    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9qdtz" podStartSLOduration=1.4241953889999999 podStartE2EDuration="1.424195389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:54:53.424167187 +0000 UTC m=+252.280020455" watchObservedRunningTime="2026-09-15 18:54:53.424195389 +0000 UTC m=+252.280048658"
Sep 15 18:54:53.433429 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.433408    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:53.433501 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.433483    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:53.433577 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:53.433561    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 18:54:53.433659 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:53.433622    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls podName:bd436ebb-24ed-4151-aa7c-c2baa894e4ff nodeName:}" failed. No retries permitted until 2026-09-15 18:54:54.433610335 +0000 UTC m=+253.289463587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-58mhq" (UID: "bd436ebb-24ed-4151-aa7c-c2baa894e4ff") : secret "cluster-monitoring-operator-tls" not found
Sep 15 18:54:53.433705 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:53.433567    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 18:54:53.433705 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:53.433701    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls podName:adf1bcc7-1575-44ed-a4c7-273bf659fc11 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:54.433689309 +0000 UTC m=+253.289542560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vmdj5" (UID: "adf1bcc7-1575-44ed-a4c7-273bf659fc11") : secret "samples-operator-tls" not found
Sep 15 18:54:53.534024 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:53.533991    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:54:53.534167 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:53.534108    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:54:53.534206 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:53.534172    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs podName:0724acd2-fe1b-4134-817d-a4fc7300ce5c nodeName:}" failed. No retries permitted until 2026-09-15 18:56:55.534154739 +0000 UTC m=+374.390007990 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs") pod "network-metrics-daemon-hfjb8" (UID: "0724acd2-fe1b-4134-817d-a4fc7300ce5c") : secret "metrics-daemon-secret" not found
Sep 15 18:54:54.445413 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:54.445372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:54.445928 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:54.445490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:54.445928 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:54.445616    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 18:54:54.445928 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:54.445695    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls podName:adf1bcc7-1575-44ed-a4c7-273bf659fc11 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:56.445676269 +0000 UTC m=+255.301529520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vmdj5" (UID: "adf1bcc7-1575-44ed-a4c7-273bf659fc11") : secret "samples-operator-tls" not found
Sep 15 18:54:54.446125 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:54.446081    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 18:54:54.446125 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:54.446122    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls podName:bd436ebb-24ed-4151-aa7c-c2baa894e4ff nodeName:}" failed. No retries permitted until 2026-09-15 18:54:56.446110422 +0000 UTC m=+255.301963670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-58mhq" (UID: "bd436ebb-24ed-4151-aa7c-c2baa894e4ff") : secret "cluster-monitoring-operator-tls" not found
Sep 15 18:54:55.425504 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:55.425468    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj" event={"ID":"3d82b329-5733-4457-b3c1-2171f98e9ff8","Type":"ContainerStarted","Data":"dc91a8ade5d20bc2151d4ee78a5489cc978f8f9760aef93c7f194c89e354e23f"}
Sep 15 18:54:55.439386 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:55.439336    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-shdhj" podStartSLOduration=2.113488574 podStartE2EDuration="3.439188922s" totalImagesPullingTime="1.325700348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:54:55.439057086 +0000 UTC m=+254.294910355" watchObservedRunningTime="2026-09-15 18:54:55.439188922 +0000 UTC m=+254.295042191"
Sep 15 18:54:56.461918 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:56.461890    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:54:56.462238 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:56.462021    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:54:56.462238 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:56.462042    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 18:54:56.462238 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:56.462118    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls podName:bd436ebb-24ed-4151-aa7c-c2baa894e4ff nodeName:}" failed. No retries permitted until 2026-09-15 18:55:00.462097089 +0000 UTC m=+259.317950340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-58mhq" (UID: "bd436ebb-24ed-4151-aa7c-c2baa894e4ff") : secret "cluster-monitoring-operator-tls" not found
Sep 15 18:54:56.462238 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:56.462148    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 18:54:56.462238 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:56.462219    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls podName:adf1bcc7-1575-44ed-a4c7-273bf659fc11 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:00.462201315 +0000 UTC m=+259.318054567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vmdj5" (UID: "adf1bcc7-1575-44ed-a4c7-273bf659fc11") : secret "samples-operator-tls" not found
Sep 15 18:54:57.433145 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.433095    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-8n8xn" event={"ID":"f470421a-2714-4f71-9eb4-f2365287496a","Type":"ContainerStarted","Data":"9a7201afe5fabba4c29c7e3389a5814d5618f34d5bdfbaf8f47c1151ee77a856"}
Sep 15 18:54:57.434571 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.434545    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw" event={"ID":"27cc90e0-3da1-4615-96f3-fa7b0f89488b","Type":"ContainerStarted","Data":"c0db88ec658035478e69a660b804fec17b9aec9a40970605d2f1fd33074849c2"}
Sep 15 18:54:57.436306 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.436282    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/0.log"
Sep 15 18:54:57.436426 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.436315    2556 generic.go:356] "Generic (PLEG): container finished" podID="270f6947-813e-465b-9ad8-8428cf35b988" containerID="4f55bc572915ae0dfeaa3ffebdeaf853a356e83b3cd834fcb186b0aff9acb28f" exitCode=255
Sep 15 18:54:57.436426 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.436350    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9" event={"ID":"270f6947-813e-465b-9ad8-8428cf35b988","Type":"ContainerDied","Data":"4f55bc572915ae0dfeaa3ffebdeaf853a356e83b3cd834fcb186b0aff9acb28f"}
Sep 15 18:54:57.436602 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.436585    2556 scope.go:117] "RemoveContainer" containerID="4f55bc572915ae0dfeaa3ffebdeaf853a356e83b3cd834fcb186b0aff9acb28f"
Sep 15 18:54:57.449887 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.449834    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-8n8xn" podStartSLOduration=2.181295096 podStartE2EDuration="5.44981412s" totalImagesPullingTime="3.268519024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:54:57.447620907 +0000 UTC m=+256.303474176" watchObservedRunningTime="2026-09-15 18:54:57.44981412 +0000 UTC m=+256.305667388"
Sep 15 18:54:57.463060 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.463012    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw" podStartSLOduration=2.125448828 podStartE2EDuration="5.462993057s" totalImagesPullingTime="3.337544229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:54:57.461811719 +0000 UTC m=+256.317664990" watchObservedRunningTime="2026-09-15 18:54:57.462993057 +0000 UTC m=+256.318846327"
Sep 15 18:54:57.895728 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.895693    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq"]
Sep 15 18:54:57.898526 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.898510    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq"
Sep 15 18:54:57.900788 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.900763    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 18:54:57.901134 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.901111    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qh5q8\""
Sep 15 18:54:57.901229 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.901148    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 18:54:57.906291 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.906269    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq"]
Sep 15 18:54:57.976743 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:57.976718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zzh\" (UniqueName: \"kubernetes.io/projected/d819910a-7306-41e9-ae7a-1eb057d41fad-kube-api-access-l6zzh\") pod \"migrator-8b4d697c5-hc2vq\" (UID: \"d819910a-7306-41e9-ae7a-1eb057d41fad\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq"
Sep 15 18:54:58.078112 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.078077    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zzh\" (UniqueName: \"kubernetes.io/projected/d819910a-7306-41e9-ae7a-1eb057d41fad-kube-api-access-l6zzh\") pod \"migrator-8b4d697c5-hc2vq\" (UID: \"d819910a-7306-41e9-ae7a-1eb057d41fad\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq"
Sep 15 18:54:58.087850 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.087766    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zzh\" (UniqueName: \"kubernetes.io/projected/d819910a-7306-41e9-ae7a-1eb057d41fad-kube-api-access-l6zzh\") pod \"migrator-8b4d697c5-hc2vq\" (UID: \"d819910a-7306-41e9-ae7a-1eb057d41fad\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq"
Sep 15 18:54:58.167749 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.167677    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dqshd"]
Sep 15 18:54:58.170856 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.170834    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.173344 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.173296    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 18:54:58.173344 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.173329    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tvqm8\""
Sep 15 18:54:58.173603 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.173588    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 18:54:58.179094 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.179073    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-data-volume\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.179170 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.179108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.179262 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.179243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.179307 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.179291    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-crio-socket\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.179341 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.179314    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqdd\" (UniqueName: \"kubernetes.io/projected/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-api-access-lrqdd\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.184395 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.184376    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dqshd"]
Sep 15 18:54:58.208239 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.208219    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq"
Sep 15 18:54:58.279787 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.279762    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-data-volume\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.279910 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.279791    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.279910 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.279857    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.279910 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.279890    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-crio-socket\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.279910 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.279905    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqdd\" (UniqueName: \"kubernetes.io/projected/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-api-access-lrqdd\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.280065 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.279950    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.280065 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.280014    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:58.779998453 +0000 UTC m=+257.635851698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.280065 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.280058    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-data-volume\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.280168 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.280138    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.280168 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.280155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-crio-socket\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.280240 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.280177    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:58.780165716 +0000 UTC m=+257.636018963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.290072 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.290051    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqdd\" (UniqueName: \"kubernetes.io/projected/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-api-access-lrqdd\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.441257 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.441184    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 18:54:58.441599 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.441583    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/0.log"
Sep 15 18:54:58.441685 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.441617    2556 generic.go:356] "Generic (PLEG): container finished" podID="270f6947-813e-465b-9ad8-8428cf35b988" containerID="2512ae66b76594855fc9c73339825bfad5a08c14b83bc314e849337771096ab5" exitCode=255
Sep 15 18:54:58.441685 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.441662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9" event={"ID":"270f6947-813e-465b-9ad8-8428cf35b988","Type":"ContainerDied","Data":"2512ae66b76594855fc9c73339825bfad5a08c14b83bc314e849337771096ab5"}
Sep 15 18:54:58.441781 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.441722    2556 scope.go:117] "RemoveContainer" containerID="4f55bc572915ae0dfeaa3ffebdeaf853a356e83b3cd834fcb186b0aff9acb28f"
Sep 15 18:54:58.442012 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.441989    2556 scope.go:117] "RemoveContainer" containerID="2512ae66b76594855fc9c73339825bfad5a08c14b83bc314e849337771096ab5"
Sep 15 18:54:58.442191 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.442170    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-g4gm9_openshift-console-operator(270f6947-813e-465b-9ad8-8428cf35b988)\"" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9" podUID="270f6947-813e-465b-9ad8-8428cf35b988"
Sep 15 18:54:58.476493 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.476473    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq"]
Sep 15 18:54:58.478990 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:54:58.478965    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd819910a_7306_41e9_ae7a_1eb057d41fad.slice/crio-06b9c4709e0a8400d57cc4c144de67bdf404c174bc7206b52eaffd8611531865 WatchSource:0}: Error finding container 06b9c4709e0a8400d57cc4c144de67bdf404c174bc7206b52eaffd8611531865: Status 404 returned error can't find the container with id 06b9c4709e0a8400d57cc4c144de67bdf404c174bc7206b52eaffd8611531865
Sep 15 18:54:58.783821 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.783742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.783821 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:58.783802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:58.783999 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.783879    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.783999 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.783898    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.783999 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.783941    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:59.783924806 +0000 UTC m=+258.639778051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.783999 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:58.783955    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:59.783949672 +0000 UTC m=+258.639802917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : configmap "kube-rbac-proxy" not found
Sep 15 18:54:59.446444 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:59.446407    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq" event={"ID":"d819910a-7306-41e9-ae7a-1eb057d41fad","Type":"ContainerStarted","Data":"06b9c4709e0a8400d57cc4c144de67bdf404c174bc7206b52eaffd8611531865"}
Sep 15 18:54:59.447980 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:59.447955    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 18:54:59.448610 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:59.448326    2556 scope.go:117] "RemoveContainer" containerID="2512ae66b76594855fc9c73339825bfad5a08c14b83bc314e849337771096ab5"
Sep 15 18:54:59.448610 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:59.448523    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-g4gm9_openshift-console-operator(270f6947-813e-465b-9ad8-8428cf35b988)\"" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9" podUID="270f6947-813e-465b-9ad8-8428cf35b988"
Sep 15 18:54:59.792903 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:59.792879    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:59.793220 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:54:59.792940    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:54:59.793220 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:59.793003    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:59.793220 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:59.793041    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:01.793028874 +0000 UTC m=+260.648882120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : configmap "kube-rbac-proxy" not found
Sep 15 18:54:59.793220 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:59.793044    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:59.793220 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:54:59.793106    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:01.793089407 +0000 UTC m=+260.648942667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:00.453912 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:00.453882    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq" event={"ID":"d819910a-7306-41e9-ae7a-1eb057d41fad","Type":"ContainerStarted","Data":"04ad16dbb458e5f78e3b9d6badef1a8b0fabd1b116423a156c8875a3799c6db1"}
Sep 15 18:55:00.454078 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:00.453933    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq" event={"ID":"d819910a-7306-41e9-ae7a-1eb057d41fad","Type":"ContainerStarted","Data":"278582738a195702b3d0a500061cda0e424891b0e3c553903eaabcb83f3a93f4"}
Sep 15 18:55:00.469322 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:00.469279    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hc2vq" podStartSLOduration=2.224234125 podStartE2EDuration="3.469262709s" totalImagesPullingTime="1.245028584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:00.467619362 +0000 UTC m=+259.323472631" watchObservedRunningTime="2026-09-15 18:55:00.469262709 +0000 UTC m=+259.325116038"
Sep 15 18:55:00.498958 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:00.498930    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:55:00.499062 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:00.499037    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:55:00.499109 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:00.499073    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 18:55:00.499146 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:00.499126    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls podName:bd436ebb-24ed-4151-aa7c-c2baa894e4ff nodeName:}" failed. No retries permitted until 2026-09-15 18:55:08.499113016 +0000 UTC m=+267.354966262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-58mhq" (UID: "bd436ebb-24ed-4151-aa7c-c2baa894e4ff") : secret "cluster-monitoring-operator-tls" not found
Sep 15 18:55:00.499196 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:00.499160    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 18:55:00.499232 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:00.499217    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls podName:adf1bcc7-1575-44ed-a4c7-273bf659fc11 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:08.499201765 +0000 UTC m=+267.355055016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-vmdj5" (UID: "adf1bcc7-1575-44ed-a4c7-273bf659fc11") : secret "samples-operator-tls" not found
Sep 15 18:55:00.895812 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:00.895788    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-64626_09057b3f-4b9b-4839-a4c4-f37bced3b634/dns-node-resolver/0.log"
Sep 15 18:55:01.810654 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:01.810595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:01.810842 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:01.810721    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:55:01.810842 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:01.810758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:01.810922 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:01.810847    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:05.810830505 +0000 UTC m=+264.666683751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : configmap "kube-rbac-proxy" not found
Sep 15 18:55:01.810922 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:01.810900    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:01.810991 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:01.810963    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:05.810949719 +0000 UTC m=+264.666802969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:02.099363 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:02.099289    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sx76p_275b46c7-18c0-4a76-8af3-1b6f17b2fbec/node-ca/0.log"
Sep 15 18:55:02.898088 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:02.898061    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5k94_300c7133-fd1a-4a29-95da-f0ebaaf366da/serve-healthcheck-canary/0.log"
Sep 15 18:55:02.989145 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:02.989108    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:55:02.989294 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:02.989170    2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:55:02.989499 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:02.989486    2556 scope.go:117] "RemoveContainer" containerID="2512ae66b76594855fc9c73339825bfad5a08c14b83bc314e849337771096ab5"
Sep 15 18:55:02.989669 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:02.989630    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-g4gm9_openshift-console-operator(270f6947-813e-465b-9ad8-8428cf35b988)\"" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9" podUID="270f6947-813e-465b-9ad8-8428cf35b988"
Sep 15 18:55:03.096137 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:03.096104    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hc2vq_d819910a-7306-41e9-ae7a-1eb057d41fad/migrator/0.log"
Sep 15 18:55:03.296222 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:03.296149    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hc2vq_d819910a-7306-41e9-ae7a-1eb057d41fad/graceful-termination/0.log"
Sep 15 18:55:05.847233 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:05.847131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:05.847612 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:05.847239    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:05.847612 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:05.847284    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:05.847612 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:05.847358    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:13.847341916 +0000 UTC m=+272.703195162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:05.847612 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:05.847359    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:55:05.847612 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:05.847388    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:13.847381069 +0000 UTC m=+272.703234315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : configmap "kube-rbac-proxy" not found
Sep 15 18:55:08.572109 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:08.572075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:55:08.572505 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:08.572214    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 18:55:08.572505 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:08.572234    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:55:08.572505 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:08.572282    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls podName:bd436ebb-24ed-4151-aa7c-c2baa894e4ff nodeName:}" failed. No retries permitted until 2026-09-15 18:55:24.572267416 +0000 UTC m=+283.428120680 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-58mhq" (UID: "bd436ebb-24ed-4151-aa7c-c2baa894e4ff") : secret "cluster-monitoring-operator-tls" not found
Sep 15 18:55:08.574499 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:08.574479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/adf1bcc7-1575-44ed-a4c7-273bf659fc11-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-vmdj5\" (UID: \"adf1bcc7-1575-44ed-a4c7-273bf659fc11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:55:08.582357 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:08.582332    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"
Sep 15 18:55:08.846430 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:08.846262    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5"]
Sep 15 18:55:09.482435 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:09.482393    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5" event={"ID":"adf1bcc7-1575-44ed-a4c7-273bf659fc11","Type":"ContainerStarted","Data":"543e82b4bf30167b263a5abe9b29837144b5a34442436a669dfe5ee4ca0e567a"}
Sep 15 18:55:11.490968 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:11.490931    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5" event={"ID":"adf1bcc7-1575-44ed-a4c7-273bf659fc11","Type":"ContainerStarted","Data":"80be2cc4adb5e7632ebb80ae0093acd29241504276730f3b412b9aa4fd39d984"}
Sep 15 18:55:11.491373 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:11.490980    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5" event={"ID":"adf1bcc7-1575-44ed-a4c7-273bf659fc11","Type":"ContainerStarted","Data":"d751dd5d9888db47e320ef0ffd52ee0ba339778875c61be1c39fb8e79e43ce5c"}
Sep 15 18:55:11.506521 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:11.506472    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-vmdj5" podStartSLOduration=17.948642926 podStartE2EDuration="19.506455698s" totalImagesPullingTime="1.557812772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:11.505737339 +0000 UTC m=+270.361590618" watchObservedRunningTime="2026-09-15 18:55:11.506455698 +0000 UTC m=+270.362308966"
Sep 15 18:55:13.915967 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:13.915931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:13.916351 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:13.915990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:13.916351 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:13.916096    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:13.916351 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:13.916170    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls podName:aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:29.916153871 +0000 UTC m=+288.772007117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqshd" (UID: "aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:13.916458 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:13.916442    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:15.672745 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:15.672718    2556 scope.go:117] "RemoveContainer" containerID="2512ae66b76594855fc9c73339825bfad5a08c14b83bc314e849337771096ab5"
Sep 15 18:55:16.509629 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:16.509604    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 18:55:16.509806 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:16.509704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9" event={"ID":"270f6947-813e-465b-9ad8-8428cf35b988","Type":"ContainerStarted","Data":"6bd7327096a22800363d173a54c8aef3c564ffe6be986b91a90f1cc2d16a7fb7"}
Sep 15 18:55:16.509996 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:16.509966    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:55:16.526428 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:16.526384    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9" podStartSLOduration=21.295421 podStartE2EDuration="24.526369186s" totalImagesPullingTime="3.230948186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:16.525347019 +0000 UTC m=+275.381200288" watchObservedRunningTime="2026-09-15 18:55:16.526369186 +0000 UTC m=+275.382222454"
Sep 15 18:55:16.651818 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:16.651793    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-g4gm9"
Sep 15 18:55:22.189758 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:22.189720    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl" podUID="1fe06272-485c-452f-8eda-13dfb438a19a"
Sep 15 18:55:22.189758 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:22.189720    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jf5l9" podUID="cdaba30b-50d4-422c-88d4-ac0f3dfb01c8"
Sep 15 18:55:22.482596 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.482510    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-qb526"]
Sep 15 18:55:22.485487 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.485467    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qb526"
Sep 15 18:55:22.487849 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.487829    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xckkr\""
Sep 15 18:55:22.487972 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.487854    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 18:55:22.487972 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.487874    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 18:55:22.496033 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.496014    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qb526"]
Sep 15 18:55:22.528736 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.528708    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:55:22.528847 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.528708    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:55:22.584848 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.584805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r64kq\" (UniqueName: \"kubernetes.io/projected/d55b7ee3-bbf5-4dba-b743-964546e2bdeb-kube-api-access-r64kq\") pod \"downloads-76f99584f4-qb526\" (UID: \"d55b7ee3-bbf5-4dba-b743-964546e2bdeb\") " pod="openshift-console/downloads-76f99584f4-qb526"
Sep 15 18:55:22.686079 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.686052    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r64kq\" (UniqueName: \"kubernetes.io/projected/d55b7ee3-bbf5-4dba-b743-964546e2bdeb-kube-api-access-r64kq\") pod \"downloads-76f99584f4-qb526\" (UID: \"d55b7ee3-bbf5-4dba-b743-964546e2bdeb\") " pod="openshift-console/downloads-76f99584f4-qb526"
Sep 15 18:55:22.694365 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.694343    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r64kq\" (UniqueName: \"kubernetes.io/projected/d55b7ee3-bbf5-4dba-b743-964546e2bdeb-kube-api-access-r64kq\") pod \"downloads-76f99584f4-qb526\" (UID: \"d55b7ee3-bbf5-4dba-b743-964546e2bdeb\") " pod="openshift-console/downloads-76f99584f4-qb526"
Sep 15 18:55:22.793666 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:22.793562    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qb526"
Sep 15 18:55:23.065750 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:23.065669    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qb526"]
Sep 15 18:55:23.069892 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:23.069863    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55b7ee3_bbf5_4dba_b743_964546e2bdeb.slice/crio-6837695c522b22dec21f0d7cac8bdabbeaa5869462dcb076cefadcf50980ddc5 WatchSource:0}: Error finding container 6837695c522b22dec21f0d7cac8bdabbeaa5869462dcb076cefadcf50980ddc5: Status 404 returned error can't find the container with id 6837695c522b22dec21f0d7cac8bdabbeaa5869462dcb076cefadcf50980ddc5
Sep 15 18:55:23.533189 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:23.533152    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qb526" event={"ID":"d55b7ee3-bbf5-4dba-b743-964546e2bdeb","Type":"ContainerStarted","Data":"6837695c522b22dec21f0d7cac8bdabbeaa5869462dcb076cefadcf50980ddc5"}
Sep 15 18:55:24.600242 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:24.600207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:55:24.602608 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:24.602584    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd436ebb-24ed-4151-aa7c-c2baa894e4ff-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-58mhq\" (UID: \"bd436ebb-24ed-4151-aa7c-c2baa894e4ff\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:55:24.775374 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:24.775341    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"
Sep 15 18:55:25.051568 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:25.051535    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq"]
Sep 15 18:55:25.054295 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:25.054271    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd436ebb_24ed_4151_aa7c_c2baa894e4ff.slice/crio-8d7ea0d797b889e7ee37fd29e89671a298e2719aab747c4644e41809c0f1f33d WatchSource:0}: Error finding container 8d7ea0d797b889e7ee37fd29e89671a298e2719aab747c4644e41809c0f1f33d: Status 404 returned error can't find the container with id 8d7ea0d797b889e7ee37fd29e89671a298e2719aab747c4644e41809c0f1f33d
Sep 15 18:55:25.541999 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:25.541960    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq" event={"ID":"bd436ebb-24ed-4151-aa7c-c2baa894e4ff","Type":"ContainerStarted","Data":"8d7ea0d797b889e7ee37fd29e89671a298e2719aab747c4644e41809c0f1f33d"}
Sep 15 18:55:26.014052 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.014008    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:55:26.016956 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.016888    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"image-registry-76c555597d-46l4z\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") " pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:55:26.115415 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.115378    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:55:26.115570 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.115536    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:55:26.118219 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.118194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdaba30b-50d4-422c-88d4-ac0f3dfb01c8-metrics-tls\") pod \"dns-default-jf5l9\" (UID: \"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8\") " pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:55:26.119033 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.118987    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fe06272-485c-452f-8eda-13dfb438a19a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p4frl\" (UID: \"1fe06272-485c-452f-8eda-13dfb438a19a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:55:26.132112 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.132089    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rr57c\""
Sep 15 18:55:26.132780 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.132758    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hj774\""
Sep 15 18:55:26.140513 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.140494    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"
Sep 15 18:55:26.140618 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.140524    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:55:26.274955 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.274882    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwrs6\""
Sep 15 18:55:26.282807 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.282773    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:55:26.433776 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.433693    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jf5l9"]
Sep 15 18:55:26.435346 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:26.435321    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl"]
Sep 15 18:55:26.850564 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:26.850530    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe06272_485c_452f_8eda_13dfb438a19a.slice/crio-608573cd1b8a5dcf088832daae89bf3cfedabd3d89c1f80d1a42e7c0da703249 WatchSource:0}: Error finding container 608573cd1b8a5dcf088832daae89bf3cfedabd3d89c1f80d1a42e7c0da703249: Status 404 returned error can't find the container with id 608573cd1b8a5dcf088832daae89bf3cfedabd3d89c1f80d1a42e7c0da703249
Sep 15 18:55:26.851387 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:26.851346    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdaba30b_50d4_422c_88d4_ac0f3dfb01c8.slice/crio-285f30c513a6d550d5413366999a3ca04b9179fe466e3a66e3b9e156eb5f526b WatchSource:0}: Error finding container 285f30c513a6d550d5413366999a3ca04b9179fe466e3a66e3b9e156eb5f526b: Status 404 returned error can't find the container with id 285f30c513a6d550d5413366999a3ca04b9179fe466e3a66e3b9e156eb5f526b
Sep 15 18:55:27.128536 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.128506    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c555597d-46l4z"]
Sep 15 18:55:27.132119 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:27.132082    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66c4f4a_c7e9_4a1f_9313_721feeee1c90.slice/crio-e6ba396976d7efd8957c5333397b0978107322ffbe0af5de40825f4360fcd8e2 WatchSource:0}: Error finding container e6ba396976d7efd8957c5333397b0978107322ffbe0af5de40825f4360fcd8e2: Status 404 returned error can't find the container with id e6ba396976d7efd8957c5333397b0978107322ffbe0af5de40825f4360fcd8e2
Sep 15 18:55:27.551113 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.550815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq" event={"ID":"bd436ebb-24ed-4151-aa7c-c2baa894e4ff","Type":"ContainerStarted","Data":"8132741d6e42d318b39e38ca5642599e810de0aa05228f823c6760858ddc5b55"}
Sep 15 18:55:27.552325 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.552288    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl" event={"ID":"1fe06272-485c-452f-8eda-13dfb438a19a","Type":"ContainerStarted","Data":"608573cd1b8a5dcf088832daae89bf3cfedabd3d89c1f80d1a42e7c0da703249"}
Sep 15 18:55:27.554356 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.554143    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c555597d-46l4z" event={"ID":"e66c4f4a-c7e9-4a1f-9313-721feeee1c90","Type":"ContainerStarted","Data":"2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed"}
Sep 15 18:55:27.554356 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.554204    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c555597d-46l4z" event={"ID":"e66c4f4a-c7e9-4a1f-9313-721feeee1c90","Type":"ContainerStarted","Data":"e6ba396976d7efd8957c5333397b0978107322ffbe0af5de40825f4360fcd8e2"}
Sep 15 18:55:27.554356 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.554247    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:55:27.555685 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.555660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jf5l9" event={"ID":"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8","Type":"ContainerStarted","Data":"285f30c513a6d550d5413366999a3ca04b9179fe466e3a66e3b9e156eb5f526b"}
Sep 15 18:55:27.565937 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.565891    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-58mhq" podStartSLOduration=33.723140339 podStartE2EDuration="35.56587345s" totalImagesPullingTime="1.842733111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:27.564965256 +0000 UTC m=+286.420818523" watchObservedRunningTime="2026-09-15 18:55:27.56587345 +0000 UTC m=+286.421726720"
Sep 15 18:55:27.592849 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:27.592799    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76c555597d-46l4z" podStartSLOduration=285.592781945 podStartE2EDuration="4m45.592781945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:27.592171647 +0000 UTC m=+286.448024960" watchObservedRunningTime="2026-09-15 18:55:27.592781945 +0000 UTC m=+286.448635213"
Sep 15 18:55:28.560933 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:28.560897    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl" event={"ID":"1fe06272-485c-452f-8eda-13dfb438a19a","Type":"ContainerStarted","Data":"76222a103eaaa7e5c0283b5643c7d8cf890219c3af7287d5e1a94e406d31c9ec"}
Sep 15 18:55:28.577404 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:28.577347    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p4frl" podStartSLOduration=274.086972926 podStartE2EDuration="4m35.57732528s" totalImagesPullingTime="1.490352354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:28.575829927 +0000 UTC m=+287.431683196" watchObservedRunningTime="2026-09-15 18:55:28.57732528 +0000 UTC m=+287.433178544"
Sep 15 18:55:29.951496 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:29.951458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:29.954102 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:29.954074    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqshd\" (UID: \"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27\") " pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:29.980711 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:29.980688    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqshd"
Sep 15 18:55:30.281190 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:30.281108    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dqshd"]
Sep 15 18:55:30.286272 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:30.286242    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5a7c3f_9f9a_46cd_808a_7857ae8d4d27.slice/crio-dde6b058fa680ae44f4e6ef9ef89b4ef9eb040983b1101f2ee8b1a02b6f8ea7c WatchSource:0}: Error finding container dde6b058fa680ae44f4e6ef9ef89b4ef9eb040983b1101f2ee8b1a02b6f8ea7c: Status 404 returned error can't find the container with id dde6b058fa680ae44f4e6ef9ef89b4ef9eb040983b1101f2ee8b1a02b6f8ea7c
Sep 15 18:55:30.572064 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:30.571943    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqshd" event={"ID":"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27","Type":"ContainerStarted","Data":"ef69e0432769005ecc79381fd29c18beeebeee8a0d87396688c96674a9969600"}
Sep 15 18:55:30.572064 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:30.572049    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqshd" event={"ID":"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27","Type":"ContainerStarted","Data":"dde6b058fa680ae44f4e6ef9ef89b4ef9eb040983b1101f2ee8b1a02b6f8ea7c"}
Sep 15 18:55:32.582910 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:32.582865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqshd" event={"ID":"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27","Type":"ContainerStarted","Data":"23d42e0a206ff687af9c0b9ec15e8a7e536101fcabf447d46fbdaf8c029e25b9"}
Sep 15 18:55:32.584944 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:32.584917    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jf5l9" event={"ID":"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8","Type":"ContainerStarted","Data":"649d11bac67b9ae86a2c3fab37c907f059b61684bbf2e2132a13d3202f799ed1"}
Sep 15 18:55:32.585067 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:32.584967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jf5l9" event={"ID":"cdaba30b-50d4-422c-88d4-ac0f3dfb01c8","Type":"ContainerStarted","Data":"fbdb20de17c530a49ecaeaffe6c81505e3fe225d3da7ef74ba6a39a20ef1dcf9"}
Sep 15 18:55:32.585132 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:32.585081    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:55:32.603906 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:32.603865    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jf5l9" podStartSLOduration=251.342950567 podStartE2EDuration="4m16.603846868s" totalImagesPullingTime="5.260896301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:32.601990221 +0000 UTC m=+291.457843490" watchObservedRunningTime="2026-09-15 18:55:32.603846868 +0000 UTC m=+291.459700135"
Sep 15 18:55:34.597710 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:34.597668    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqshd" event={"ID":"aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27","Type":"ContainerStarted","Data":"2bc1b21bd82e2b5481b87d14d4fc1ec25f39967b10d322951673e0a49f083a23"}
Sep 15 18:55:34.617356 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:34.617313    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dqshd" podStartSLOduration=33.58472677 podStartE2EDuration="36.617298357s" totalImagesPullingTime="3.032571587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:54:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:34.617133788 +0000 UTC m=+293.472987060" watchObservedRunningTime="2026-09-15 18:55:34.617298357 +0000 UTC m=+293.473151624"
Sep 15 18:55:35.673364 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:35.673311    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hfjb8" podUID="0724acd2-fe1b-4134-817d-a4fc7300ce5c"
Sep 15 18:55:35.810189 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.810148    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g8n6m"]
Sep 15 18:55:35.813675 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.813628    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"]
Sep 15 18:55:35.813796 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.813771    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.817098 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.817075    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:35.817252 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.817105    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 18:55:35.821531 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.821509    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 18:55:35.821824 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.821807    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xf85x\""
Sep 15 18:55:35.822832 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.822733    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 18:55:35.823196 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.822990    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 18:55:35.823533 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.823508    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 18:55:35.823904 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.823886    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 18:55:35.827193 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.827172    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jdxlm\""
Sep 15 18:55:35.827616 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.827597    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 18:55:35.839737 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.839716    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"]
Sep 15 18:55:35.902186 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902154    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-textfile\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902338 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902338 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-root\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902338 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902281    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-metrics-client-ca\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902338 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902324    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7kh\" (UniqueName: \"kubernetes.io/projected/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-kube-api-access-vn7kh\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902532 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902352    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d135cf38-92b1-465f-8353-6bf796fdebe9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:35.902532 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-wtmp\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902532 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902435    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-sys\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902532 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902467    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902532 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902489    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vl7\" (UniqueName: \"kubernetes.io/projected/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-api-access-68vl7\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:35.902532 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902524    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:35.902825 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902589    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-tls\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:35.902825 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902633    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d135cf38-92b1-465f-8353-6bf796fdebe9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:35.902825 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902673    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:35.902825 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:35.902694    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.003748 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003666    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-root\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.003748 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-metrics-client-ca\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.003748 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003749    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7kh\" (UniqueName: \"kubernetes.io/projected/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-kube-api-access-vn7kh\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.003985 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d135cf38-92b1-465f-8353-6bf796fdebe9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.003985 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003798    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-wtmp\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.003985 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003800    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-root\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.003985 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-sys\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.003985 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003878    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.003985 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003911    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vl7\" (UniqueName: \"kubernetes.io/projected/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-api-access-68vl7\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.003985 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003941    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.003985 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.003952    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-wtmp\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.004342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-tls\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.004342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004050    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d135cf38-92b1-465f-8353-6bf796fdebe9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.004342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.004342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004109    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.004342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-textfile\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.004342 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.004342 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:36.004190    2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 18:55:36.004342 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:36.004267    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-tls podName:c2f1975f-6fc8-42e7-b359-ed338e9bfcb7 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:36.504246629 +0000 UTC m=+295.360099889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-tls") pod "node-exporter-g8n6m" (UID: "c2f1975f-6fc8-42e7-b359-ed338e9bfcb7") : secret "node-exporter-tls" not found
Sep 15 18:55:36.004748 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:36.004377    2556 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 15 18:55:36.004748 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004384    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-metrics-client-ca\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.004748 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-sys\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.004748 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:36.004445    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-tls podName:d135cf38-92b1-465f-8353-6bf796fdebe9 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:36.504428958 +0000 UTC m=+295.360282207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-lr5qw" (UID: "d135cf38-92b1-465f-8353-6bf796fdebe9") : secret "kube-state-metrics-tls" not found
Sep 15 18:55:36.004748 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004606    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.004748 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d135cf38-92b1-465f-8353-6bf796fdebe9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.005054 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.004792    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-textfile\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.005054 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.005020    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d135cf38-92b1-465f-8353-6bf796fdebe9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.005211 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.005169    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.007029 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.007003    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.007300 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.007282    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.012133 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.012110    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vl7\" (UniqueName: \"kubernetes.io/projected/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-api-access-68vl7\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.012318 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.012297    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7kh\" (UniqueName: \"kubernetes.io/projected/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-kube-api-access-vn7kh\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.509268 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.509217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-tls\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.509460 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.509278    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.512020 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.511990    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d135cf38-92b1-465f-8353-6bf796fdebe9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lr5qw\" (UID: \"d135cf38-92b1-465f-8353-6bf796fdebe9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.512153 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.512018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2f1975f-6fc8-42e7-b359-ed338e9bfcb7-node-exporter-tls\") pod \"node-exporter-g8n6m\" (UID: \"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7\") " pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.727494 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.727457    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g8n6m"
Sep 15 18:55:36.732813 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.732792    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"
Sep 15 18:55:36.862495 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.862407    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:55:36.866363 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.866336    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:36.868911 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.868759    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 18:55:36.868911 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.868776    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 18:55:36.868911 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.868809    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 18:55:36.868911 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.868834    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7pzzg\""
Sep 15 18:55:36.868911 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.868866    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 18:55:36.869227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.869056    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 18:55:36.869227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.869063    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 18:55:36.869227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.869073    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 18:55:36.869227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.869122    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 18:55:36.869227 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.869056    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 18:55:36.878825 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:36.878801    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:55:37.013722 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.013685    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.013900 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.013745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6c7\" (UniqueName: \"kubernetes.io/projected/6d041a31-b45b-4010-b40a-650daf542361-kube-api-access-fb6c7\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.013900 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.013779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.013900 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.013871    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014046 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.013924    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014046 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.013963    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d041a31-b45b-4010-b40a-650daf542361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014046 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.013992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014046 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.014023    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d041a31-b45b-4010-b40a-650daf542361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014185 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.014045    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d041a31-b45b-4010-b40a-650daf542361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014185 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.014098    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014185 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.014122    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d041a31-b45b-4010-b40a-650daf542361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014299 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.014212    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d041a31-b45b-4010-b40a-650daf542361-config-out\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.014299 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.014240    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-web-config\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.115322 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.115248    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d041a31-b45b-4010-b40a-650daf542361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.115322 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.115285    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d041a31-b45b-4010-b40a-650daf542361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.115322 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.115312    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.115590 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.115365    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d041a31-b45b-4010-b40a-650daf542361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.115590 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.115446    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d041a31-b45b-4010-b40a-650daf542361-config-out\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.115590 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:37.115566    2556 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 15 18:55:37.115869 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:37.115632    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-main-tls podName:6d041a31-b45b-4010-b40a-650daf542361 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:37.615612201 +0000 UTC m=+296.471465460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6d041a31-b45b-4010-b40a-650daf542361") : secret "alertmanager-main-tls" not found
Sep 15 18:55:37.115869 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.115758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d041a31-b45b-4010-b40a-650daf542361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.115869 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.115805    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-web-config\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.115869 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.115836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.116110 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.116084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6c7\" (UniqueName: \"kubernetes.io/projected/6d041a31-b45b-4010-b40a-650daf542361-kube-api-access-fb6c7\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.116164 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.116142    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.116214 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.116197    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.116265 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.116237    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.116317 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.116292    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d041a31-b45b-4010-b40a-650daf542361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.116377 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.116323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.116437 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.116422    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d041a31-b45b-4010-b40a-650daf542361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.117665 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.117610    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d041a31-b45b-4010-b40a-650daf542361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.120056 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.119126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-web-config\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.120792 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.120099    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.120792 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.120661    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.120792 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.120727    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.120792 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.120749    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d041a31-b45b-4010-b40a-650daf542361-config-out\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.121026 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.120898    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d041a31-b45b-4010-b40a-650daf542361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.121195 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.121158    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.122485 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.122425    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.124758 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.124735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6c7\" (UniqueName: \"kubernetes.io/projected/6d041a31-b45b-4010-b40a-650daf542361-kube-api-access-fb6c7\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.621397 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.621361    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.624465 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.624120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d041a31-b45b-4010-b40a-650daf542361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d041a31-b45b-4010-b40a-650daf542361\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:37.778997 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:37.778962    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:55:40.211543 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.211507    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86f978c87-mbp62"]
Sep 15 18:55:40.215037 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.215016    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.218498 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.218422    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 18:55:40.218498 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.218425    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 18:55:40.218498 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.218449    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8eq68oq9hl3un\""
Sep 15 18:55:40.218498 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.218476    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ph4sw\""
Sep 15 18:55:40.218794 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.218427    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 18:55:40.218794 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.218430    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 18:55:40.226201 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.226176    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86f978c87-mbp62"]
Sep 15 18:55:40.246848 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.246814    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-client-ca-bundle\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.246963 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.246855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5322c9-d822-452b-8cb2-7b115ac95f05-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.246963 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.246892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2e5322c9-d822-452b-8cb2-7b115ac95f05-audit-log\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.247085 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.247000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-secret-metrics-server-client-certs\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.247085 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.247035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-secret-metrics-server-tls\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.247207 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.247095    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2e5322c9-d822-452b-8cb2-7b115ac95f05-metrics-server-audit-profiles\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.247207 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.247122    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcw96\" (UniqueName: \"kubernetes.io/projected/2e5322c9-d822-452b-8cb2-7b115ac95f05-kube-api-access-fcw96\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.348439 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.348399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-secret-metrics-server-client-certs\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.348439 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.348441    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-secret-metrics-server-tls\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.348700 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.348557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2e5322c9-d822-452b-8cb2-7b115ac95f05-metrics-server-audit-profiles\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.348700 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.348586    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcw96\" (UniqueName: \"kubernetes.io/projected/2e5322c9-d822-452b-8cb2-7b115ac95f05-kube-api-access-fcw96\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.348700 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.348633    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-client-ca-bundle\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.348700 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.348687    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5322c9-d822-452b-8cb2-7b115ac95f05-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.348898 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.348722    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2e5322c9-d822-452b-8cb2-7b115ac95f05-audit-log\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.349154 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.349097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2e5322c9-d822-452b-8cb2-7b115ac95f05-audit-log\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.349915 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.349865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5322c9-d822-452b-8cb2-7b115ac95f05-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.350021 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.349960    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2e5322c9-d822-452b-8cb2-7b115ac95f05-metrics-server-audit-profiles\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.351480 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.351457    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-secret-metrics-server-tls\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.351585 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.351464    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-client-ca-bundle\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.352117 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.352096    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2e5322c9-d822-452b-8cb2-7b115ac95f05-secret-metrics-server-client-certs\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.356449 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.356421    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcw96\" (UniqueName: \"kubernetes.io/projected/2e5322c9-d822-452b-8cb2-7b115ac95f05-kube-api-access-fcw96\") pod \"metrics-server-86f978c87-mbp62\" (UID: \"2e5322c9-d822-452b-8cb2-7b115ac95f05\") " pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.527516 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.527424    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:55:40.580367 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.580322    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"]
Sep 15 18:55:40.583530 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.583507    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:40.586253 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.586232    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 18:55:40.586353 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.586254    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mcsk5\""
Sep 15 18:55:40.592018 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.591995    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"]
Sep 15 18:55:40.600451 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:40.599872    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f1975f_6fc8_42e7_b359_ed338e9bfcb7.slice/crio-1415d114f1b060ff58676887813fd0fb16c7f7241ba73e3734bf41c58d2f5887 WatchSource:0}: Error finding container 1415d114f1b060ff58676887813fd0fb16c7f7241ba73e3734bf41c58d2f5887: Status 404 returned error can't find the container with id 1415d114f1b060ff58676887813fd0fb16c7f7241ba73e3734bf41c58d2f5887
Sep 15 18:55:40.624438 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.624402    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8n6m" event={"ID":"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7","Type":"ContainerStarted","Data":"1415d114f1b060ff58676887813fd0fb16c7f7241ba73e3734bf41c58d2f5887"}
Sep 15 18:55:40.651940 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.651672    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d67nm\" (UniqueName: \"kubernetes.io/projected/9e848b73-2271-4a1e-a017-4f1a4d156f42-kube-api-access-d67nm\") pod \"monitoring-plugin-5d85778b9d-j2hhw\" (UID: \"9e848b73-2271-4a1e-a017-4f1a4d156f42\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:40.651940 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.651738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e848b73-2271-4a1e-a017-4f1a4d156f42-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j2hhw\" (UID: \"9e848b73-2271-4a1e-a017-4f1a4d156f42\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:40.752354 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.752326    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67nm\" (UniqueName: \"kubernetes.io/projected/9e848b73-2271-4a1e-a017-4f1a4d156f42-kube-api-access-d67nm\") pod \"monitoring-plugin-5d85778b9d-j2hhw\" (UID: \"9e848b73-2271-4a1e-a017-4f1a4d156f42\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:40.752455 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.752391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e848b73-2271-4a1e-a017-4f1a4d156f42-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j2hhw\" (UID: \"9e848b73-2271-4a1e-a017-4f1a4d156f42\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:40.752557 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:40.752538    2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 15 18:55:40.752601 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:55:40.752594    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e848b73-2271-4a1e-a017-4f1a4d156f42-monitoring-plugin-cert podName:9e848b73-2271-4a1e-a017-4f1a4d156f42 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:41.252579524 +0000 UTC m=+300.108432770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9e848b73-2271-4a1e-a017-4f1a4d156f42-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-j2hhw" (UID: "9e848b73-2271-4a1e-a017-4f1a4d156f42") : secret "monitoring-plugin-cert" not found
Sep 15 18:55:40.760592 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.760563    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67nm\" (UniqueName: \"kubernetes.io/projected/9e848b73-2271-4a1e-a017-4f1a4d156f42-kube-api-access-d67nm\") pod \"monitoring-plugin-5d85778b9d-j2hhw\" (UID: \"9e848b73-2271-4a1e-a017-4f1a4d156f42\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:40.775328 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.775298    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-979cd9b44-jxqss"]
Sep 15 18:55:40.779259 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.779207    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.781482 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.781459    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 18:55:40.781584 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.781483    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 18:55:40.781584 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.781494    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 18:55:40.781902 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.781880    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kbhdv\""
Sep 15 18:55:40.781902 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.781905    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 18:55:40.782065 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.781921    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 18:55:40.786831 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.786573    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 18:55:40.789463 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.789440    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-979cd9b44-jxqss"]
Sep 15 18:55:40.853429 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.853398    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-oauth-config\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.853429 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.853436    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-console-config\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.853682 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.853458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-service-ca\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.853682 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.853485    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-serving-cert\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.853682 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.853613    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-trusted-ca-bundle\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.853831 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.853707    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47wk\" (UniqueName: \"kubernetes.io/projected/7e4dead5-80fc-4dae-9c5d-244313248689-kube-api-access-h47wk\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.853831 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.853734    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-oauth-serving-cert\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.877762 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.877731    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw"]
Sep 15 18:55:40.880630 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.880566    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86f978c87-mbp62"]
Sep 15 18:55:40.880830 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:40.880775    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd135cf38_92b1_465f_8353_6bf796fdebe9.slice/crio-44d61db066ebc807fe02202140ac5412bf6dc7453f03696a4c8ac25109e8dc15 WatchSource:0}: Error finding container 44d61db066ebc807fe02202140ac5412bf6dc7453f03696a4c8ac25109e8dc15: Status 404 returned error can't find the container with id 44d61db066ebc807fe02202140ac5412bf6dc7453f03696a4c8ac25109e8dc15
Sep 15 18:55:40.883782 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.883533    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:55:40.883993 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:40.883944    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5322c9_d822_452b_8cb2_7b115ac95f05.slice/crio-cc930c23ef9b29f812d9b4fb4bd1920a283a6cfc5cb3491a1b31ae57d834e1a5 WatchSource:0}: Error finding container cc930c23ef9b29f812d9b4fb4bd1920a283a6cfc5cb3491a1b31ae57d834e1a5: Status 404 returned error can't find the container with id cc930c23ef9b29f812d9b4fb4bd1920a283a6cfc5cb3491a1b31ae57d834e1a5
Sep 15 18:55:40.889154 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:40.889131    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d041a31_b45b_4010_b40a_650daf542361.slice/crio-5d4db5bb33699f7a7e6e6bb7bcbf2c330c8779aa21dcd4dac2a469f8e66d9690 WatchSource:0}: Error finding container 5d4db5bb33699f7a7e6e6bb7bcbf2c330c8779aa21dcd4dac2a469f8e66d9690: Status 404 returned error can't find the container with id 5d4db5bb33699f7a7e6e6bb7bcbf2c330c8779aa21dcd4dac2a469f8e66d9690
Sep 15 18:55:40.955003 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.954963    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-oauth-config\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.955003 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.955011    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-console-config\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.955221 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.955036    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-service-ca\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.955221 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.955086    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-serving-cert\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.955221 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.955173    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-trusted-ca-bundle\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.955373 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.955234    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47wk\" (UniqueName: \"kubernetes.io/projected/7e4dead5-80fc-4dae-9c5d-244313248689-kube-api-access-h47wk\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.955373 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.955265    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-oauth-serving-cert\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.955907 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.955852    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-console-config\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.956428 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.956382    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-trusted-ca-bundle\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.956547 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.956444    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-service-ca\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.956704 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.956673    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-oauth-serving-cert\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.958016 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.957987    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-oauth-config\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.958727 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.958704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-serving-cert\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:40.963616 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:40.963584    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47wk\" (UniqueName: \"kubernetes.io/projected/7e4dead5-80fc-4dae-9c5d-244313248689-kube-api-access-h47wk\") pod \"console-979cd9b44-jxqss\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") " pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:41.089499 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.089397    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:55:41.265137 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.265058    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e848b73-2271-4a1e-a017-4f1a4d156f42-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j2hhw\" (UID: \"9e848b73-2271-4a1e-a017-4f1a4d156f42\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:41.267966 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.267922    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e848b73-2271-4a1e-a017-4f1a4d156f42-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j2hhw\" (UID: \"9e848b73-2271-4a1e-a017-4f1a4d156f42\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:41.368349 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.368267    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-979cd9b44-jxqss"]
Sep 15 18:55:41.371532 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:41.371499    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4dead5_80fc_4dae_9c5d_244313248689.slice/crio-d9c7a766cac0de06ec658d75d8352a41e0b8fe565caa224c4f52f859db366594 WatchSource:0}: Error finding container d9c7a766cac0de06ec658d75d8352a41e0b8fe565caa224c4f52f859db366594: Status 404 returned error can't find the container with id d9c7a766cac0de06ec658d75d8352a41e0b8fe565caa224c4f52f859db366594
Sep 15 18:55:41.526805 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.526763    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:41.587846 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.587788    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 18:55:41.608455 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.603562    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 18:55:41.623702 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.623677    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 18:55:41.645272 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.636667    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 18:55:41.645272 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.637425    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d041a31-b45b-4010-b40a-650daf542361","Type":"ContainerStarted","Data":"5d4db5bb33699f7a7e6e6bb7bcbf2c330c8779aa21dcd4dac2a469f8e66d9690"}
Sep 15 18:55:41.645272 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.641815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qb526" event={"ID":"d55b7ee3-bbf5-4dba-b743-964546e2bdeb","Type":"ContainerStarted","Data":"8a4f00668bea04ac6f7f8d702e4c5f873a1f65a5625dacddba488e4fa860bd0e"}
Sep 15 18:55:41.645272 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.642498    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-qb526"
Sep 15 18:55:41.645272 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.644729    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-979cd9b44-jxqss" event={"ID":"7e4dead5-80fc-4dae-9c5d-244313248689","Type":"ContainerStarted","Data":"d9c7a766cac0de06ec658d75d8352a41e0b8fe565caa224c4f52f859db366594"}
Sep 15 18:55:41.647576 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.647457    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw" event={"ID":"d135cf38-92b1-465f-8353-6bf796fdebe9","Type":"ContainerStarted","Data":"44d61db066ebc807fe02202140ac5412bf6dc7453f03696a4c8ac25109e8dc15"}
Sep 15 18:55:41.647807 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.647737    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 18:55:41.649832 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.649809    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86f978c87-mbp62" event={"ID":"2e5322c9-d822-452b-8cb2-7b115ac95f05","Type":"ContainerStarted","Data":"cc930c23ef9b29f812d9b4fb4bd1920a283a6cfc5cb3491a1b31ae57d834e1a5"}
Sep 15 18:55:41.654322 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.654232    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-qb526"
Sep 15 18:55:41.660911 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.660827    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-qb526" podStartSLOduration=2.038630001 podStartE2EDuration="19.660809401s" totalImagesPullingTime="17.6221794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:41.659365332 +0000 UTC m=+300.515218601" watchObservedRunningTime="2026-09-15 18:55:41.660809401 +0000 UTC m=+300.516662670"
Sep 15 18:55:41.822127 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.822099    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"]
Sep 15 18:55:41.826048 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:41.826013    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e848b73_2271_4a1e_a017_4f1a4d156f42.slice/crio-293cfa5398ceb2d431abebb113570ee41ad9951e39f3575072a7a40b39360662 WatchSource:0}: Error finding container 293cfa5398ceb2d431abebb113570ee41ad9951e39f3575072a7a40b39360662: Status 404 returned error can't find the container with id 293cfa5398ceb2d431abebb113570ee41ad9951e39f3575072a7a40b39360662
Sep 15 18:55:41.939354 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.939239    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:55:41.945091 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.945063    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.948564 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.947920    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 18:55:41.948564 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.948193    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e71fuihotb006\""
Sep 15 18:55:41.949045 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.948888    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 18:55:41.950025 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.949462    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t2zfd\""
Sep 15 18:55:41.950025 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.949694    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 18:55:41.950025 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.949901    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 18:55:41.951230 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.950631    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 18:55:41.951230 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.950891    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 18:55:41.951230 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.951089    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 18:55:41.951473 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.951281    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 18:55:41.951535 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.951482    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 18:55:41.953095 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.951720    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 18:55:41.953095 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.951903    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 18:55:41.953095 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.952260    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 18:55:41.953095 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.952440    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 18:55:41.975181 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.968730    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976433    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976484    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-web-config\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976610    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976669    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-config\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3266dcc3-3962-43f6-a0e8-6661e668e406-config-out\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976795    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976833    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976886    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6nd\" (UniqueName: \"kubernetes.io/projected/3266dcc3-3962-43f6-a0e8-6661e668e406-kube-api-access-cw6nd\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976920    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.976964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.976998 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.977007    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.977737 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.977041    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.977737 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.977068    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.977737 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.977105    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3266dcc3-3962-43f6-a0e8-6661e668e406-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.977737 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.977140    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.977737 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.977175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.977737 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.977213    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:41.980928 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:41.980318    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079103    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079180    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079293    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-web-config\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079339    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079359    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-config\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3266dcc3-3962-43f6-a0e8-6661e668e406-config-out\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079465    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6nd\" (UniqueName: \"kubernetes.io/projected/3266dcc3-3962-43f6-a0e8-6661e668e406-kube-api-access-cw6nd\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079540    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079581    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079612    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079666    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079693    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.080140 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3266dcc3-3962-43f6-a0e8-6661e668e406-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.086097 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.086097 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.086097 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.079789    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.086097 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.080376    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.086097 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.080914    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.086097 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.080998    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.086097 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.085778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.094182 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.094151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.095685 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.095659    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-config\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.096397 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.096302    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3266dcc3-3962-43f6-a0e8-6661e668e406-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.097169 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.096755    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.097169 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.097129    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.097539 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.097512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3266dcc3-3962-43f6-a0e8-6661e668e406-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.099069 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.098758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3266dcc3-3962-43f6-a0e8-6661e668e406-config-out\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.099240 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.099114    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.099705 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.099520    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.100070 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.100011    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.100677 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.100172    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.100677 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.100660    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.101390 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.101066    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6nd\" (UniqueName: \"kubernetes.io/projected/3266dcc3-3962-43f6-a0e8-6661e668e406-kube-api-access-cw6nd\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.103205 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.103146    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.103916 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.103897    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-web-config\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.104313 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.104280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3266dcc3-3962-43f6-a0e8-6661e668e406-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3266dcc3-3962-43f6-a0e8-6661e668e406\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.278192 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.278113    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:42.594568 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.593712    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jf5l9"
Sep 15 18:55:42.657821 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.657783    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw" event={"ID":"9e848b73-2271-4a1e-a017-4f1a4d156f42","Type":"ContainerStarted","Data":"293cfa5398ceb2d431abebb113570ee41ad9951e39f3575072a7a40b39360662"}
Sep 15 18:55:42.813449 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:42.813323    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:55:42.928753 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:55:42.928711    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3266dcc3_3962_43f6_a0e8_6661e668e406.slice/crio-75068d919b322ab7a1805508776625c0d0f4e84c9c9c840e93c3d5ef8c80eb6b WatchSource:0}: Error finding container 75068d919b322ab7a1805508776625c0d0f4e84c9c9c840e93c3d5ef8c80eb6b: Status 404 returned error can't find the container with id 75068d919b322ab7a1805508776625c0d0f4e84c9c9c840e93c3d5ef8c80eb6b
Sep 15 18:55:43.676664 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:43.676548    2556 generic.go:356] "Generic (PLEG): container finished" podID="6d041a31-b45b-4010-b40a-650daf542361" containerID="eb05cdcda047551ef3ed9e184e89aa6451b22007af68fb701f42e32e4870ad4c" exitCode=0
Sep 15 18:55:43.680296 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:43.679389    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:55:43.681794 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:43.681545    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d041a31-b45b-4010-b40a-650daf542361","Type":"ContainerDied","Data":"eb05cdcda047551ef3ed9e184e89aa6451b22007af68fb701f42e32e4870ad4c"}
Sep 15 18:55:43.684328 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:43.683029    2556 generic.go:356] "Generic (PLEG): container finished" podID="3266dcc3-3962-43f6-a0e8-6661e668e406" containerID="e82f016133b529661f388e10f18f2e37154f9281f0158d23643c9233f7427dcd" exitCode=0
Sep 15 18:55:43.684328 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:43.684209    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3266dcc3-3962-43f6-a0e8-6661e668e406","Type":"ContainerDied","Data":"e82f016133b529661f388e10f18f2e37154f9281f0158d23643c9233f7427dcd"}
Sep 15 18:55:43.684328 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:43.684264    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3266dcc3-3962-43f6-a0e8-6661e668e406","Type":"ContainerStarted","Data":"75068d919b322ab7a1805508776625c0d0f4e84c9c9c840e93c3d5ef8c80eb6b"}
Sep 15 18:55:45.096861 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:45.096821    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76c555597d-46l4z"]
Sep 15 18:55:45.104271 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:45.104117    2556 patch_prober.go:28] interesting pod/image-registry-76c555597d-46l4z container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 18:55:45.104271 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:45.104193    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-76c555597d-46l4z" podUID="e66c4f4a-c7e9-4a1f-9313-721feeee1c90" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 18:55:49.672391 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:49.672311    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:55:51.726899 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.726827    2556 generic.go:356] "Generic (PLEG): container finished" podID="c2f1975f-6fc8-42e7-b359-ed338e9bfcb7" containerID="cc7157fd76178616ed99576b0894f0124d0a4a8869dd2411431674afe98756f7" exitCode=0
Sep 15 18:55:51.727308 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.726910    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8n6m" event={"ID":"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7","Type":"ContainerDied","Data":"cc7157fd76178616ed99576b0894f0124d0a4a8869dd2411431674afe98756f7"}
Sep 15 18:55:51.730035 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.730012    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d041a31-b45b-4010-b40a-650daf542361","Type":"ContainerStarted","Data":"8d96fdd4854fd6b0e88b9bd0abba670680745d2de99d2796d837428cbda5b42c"}
Sep 15 18:55:51.730145 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.730066    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d041a31-b45b-4010-b40a-650daf542361","Type":"ContainerStarted","Data":"5d1aebf65d22eb1ca4c3af3c483f39587c5652c5b22c3fb8892ed7b900fdd4c4"}
Sep 15 18:55:51.730145 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.730100    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d041a31-b45b-4010-b40a-650daf542361","Type":"ContainerStarted","Data":"4d89dac1794d7eb33787dd307c725b6377f634b4a6c016873aab538cd328a218"}
Sep 15 18:55:51.730145 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.730130    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d041a31-b45b-4010-b40a-650daf542361","Type":"ContainerStarted","Data":"733d9d03b4734b001c959cc9fd9f2a7472d8b98c27bf42e70d7a1c213515af03"}
Sep 15 18:55:51.730280 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.730161    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d041a31-b45b-4010-b40a-650daf542361","Type":"ContainerStarted","Data":"a36659317cd321a0ba62490ef7fd85c8ed3dfd18bf9240b560eb1ef70c9031c3"}
Sep 15 18:55:51.732206 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.732178    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3266dcc3-3962-43f6-a0e8-6661e668e406","Type":"ContainerStarted","Data":"d9b1c3c6b4eaa996b4ea6de414d3574fb205b5a733cc25ffea2dc58c50715d69"}
Sep 15 18:55:51.732304 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.732242    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3266dcc3-3962-43f6-a0e8-6661e668e406","Type":"ContainerStarted","Data":"4f5ebc337531e4af778108addfe901578aca4577300bd4f53930750dbb8ef849"}
Sep 15 18:55:51.733595 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.733570    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw" event={"ID":"9e848b73-2271-4a1e-a017-4f1a4d156f42","Type":"ContainerStarted","Data":"96e4b9968bb4e34ee1a9b5dc4e7ddd909c5163001b049adb2a7c415fd73d922f"}
Sep 15 18:55:51.733774 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.733749    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:51.735273 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.735234    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-979cd9b44-jxqss" event={"ID":"7e4dead5-80fc-4dae-9c5d-244313248689","Type":"ContainerStarted","Data":"263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7"}
Sep 15 18:55:51.737613 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.737575    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw" event={"ID":"d135cf38-92b1-465f-8353-6bf796fdebe9","Type":"ContainerStarted","Data":"4547b4f56bcd3dd1df47f6c2f897d444d3f1706b2e52c52e2453e5a44460c10b"}
Sep 15 18:55:51.737726 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.737630    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw" event={"ID":"d135cf38-92b1-465f-8353-6bf796fdebe9","Type":"ContainerStarted","Data":"95bdc0034aa6614f180afdaff2e59a9e840b84e8a00583d45242a718fde1f5ea"}
Sep 15 18:55:51.737726 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.737680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw" event={"ID":"d135cf38-92b1-465f-8353-6bf796fdebe9","Type":"ContainerStarted","Data":"d6bf01225cbfdad18286592f58970e5bae2a3d000306970507db51aebb46fe58"}
Sep 15 18:55:51.739579 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.739551    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86f978c87-mbp62" event={"ID":"2e5322c9-d822-452b-8cb2-7b115ac95f05","Type":"ContainerStarted","Data":"47b4627000abe91529c0cbbe28a2ed99b977440882c5a07174408f1598af4679"}
Sep 15 18:55:51.739707 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.739689    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw"
Sep 15 18:55:51.765030 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.764974    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2hhw" podStartSLOduration=2.7240477 podStartE2EDuration="11.764955283s" totalImagesPullingTime="9.040907583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:51.764217371 +0000 UTC m=+310.620070637" watchObservedRunningTime="2026-09-15 18:55:51.764955283 +0000 UTC m=+310.620808551"
Sep 15 18:55:51.784484 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.784441    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86f978c87-mbp62" podStartSLOduration=1.817495452 podStartE2EDuration="11.78442357s" totalImagesPullingTime="9.966928118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:51.783857046 +0000 UTC m=+310.639710315" watchObservedRunningTime="2026-09-15 18:55:51.78442357 +0000 UTC m=+310.640276840"
Sep 15 18:55:51.803067 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.803013    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-979cd9b44-jxqss" podStartSLOduration=2.289223259 podStartE2EDuration="11.802993473s" totalImagesPullingTime="9.513770214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:51.799438681 +0000 UTC m=+310.655291960" watchObservedRunningTime="2026-09-15 18:55:51.802993473 +0000 UTC m=+310.658846742"
Sep 15 18:55:51.816937 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:51.816881    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lr5qw" podStartSLOduration=6.812919549 podStartE2EDuration="16.816859804s" totalImagesPullingTime="10.003940255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:51.816177054 +0000 UTC m=+310.672030323" watchObservedRunningTime="2026-09-15 18:55:51.816859804 +0000 UTC m=+310.672713077"
Sep 15 18:55:52.747089 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:52.747050    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8n6m" event={"ID":"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7","Type":"ContainerStarted","Data":"f531539572b3f2970f6d4910779dc5d1f5023f31765a813f89178f0f51746bf0"}
Sep 15 18:55:52.747581 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:52.747151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8n6m" event={"ID":"c2f1975f-6fc8-42e7-b359-ed338e9bfcb7","Type":"ContainerStarted","Data":"e7161e30f270a4ae1abf2f95c45473694c11021da151d49662aa89ba361e990b"}
Sep 15 18:55:52.764982 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:52.764935    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g8n6m" podStartSLOduration=7.316030446 podStartE2EDuration="17.764921474s" totalImagesPullingTime="10.289069227s" totalInitContainerRuntime="159.821801ms" isStatefulPod=true podCreationTimestamp="2026-09-15 18:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:52.764134623 +0000 UTC m=+311.619987904" watchObservedRunningTime="2026-09-15 18:55:52.764921474 +0000 UTC m=+311.620774742"
Sep 15 18:55:54.763660 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:54.763601    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d041a31-b45b-4010-b40a-650daf542361","Type":"ContainerStarted","Data":"3a8725163c809cac96f0bc78a37448c3844ae2abba7753030f9d24b40c8a003b"}
Sep 15 18:55:54.767151 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:54.767123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3266dcc3-3962-43f6-a0e8-6661e668e406","Type":"ContainerStarted","Data":"a6787690f34fe09fa572e59cb1f025d5e17d090659e7298f500f58189830f667"}
Sep 15 18:55:54.767275 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:54.767190    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3266dcc3-3962-43f6-a0e8-6661e668e406","Type":"ContainerStarted","Data":"188a6a52894017b5c7cc82a6f5e0ae35a0e8177abbd3bf3e5654d701fdfd680b"}
Sep 15 18:55:54.767275 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:54.767236    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3266dcc3-3962-43f6-a0e8-6661e668e406","Type":"ContainerStarted","Data":"10ddb62e7298c382f29f2e55593ae4cb78bba4871d23b5a0f22f4894374676fd"}
Sep 15 18:55:54.767275 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:54.767271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3266dcc3-3962-43f6-a0e8-6661e668e406","Type":"ContainerStarted","Data":"378e673b492b4e14eaabaca4e4b041096c1d27d1823e7f48f76b114a41d3367c"}
Sep 15 18:55:54.787617 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:54.787573    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.513483345 podStartE2EDuration="18.787559531s" totalImagesPullingTime="11.242387294s" totalInitContainerRuntime="31.688892ms" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:54.787294836 +0000 UTC m=+313.643148116" watchObservedRunningTime="2026-09-15 18:55:54.787559531 +0000 UTC m=+313.643412798"
Sep 15 18:55:54.814153 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:54.814108    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.897474539 podStartE2EDuration="13.814092543s" totalImagesPullingTime="9.892498189s" totalInitContainerRuntime="24.119815ms" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:54.811850603 +0000 UTC m=+313.667703863" watchObservedRunningTime="2026-09-15 18:55:54.814092543 +0000 UTC m=+313.669945850"
Sep 15 18:55:55.101916 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:55.101839    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:55:57.278658 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:55:57.278597    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:56:00.527830 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:00.527795    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:56:00.528267 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:00.527874    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:56:01.090380 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:01.090344    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:56:01.090544 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:01.090409    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:56:01.095267 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:01.095243    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:56:01.799468 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:01.799436    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:56:10.127095 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.127049    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76c555597d-46l4z" podUID="e66c4f4a-c7e9-4a1f-9313-721feeee1c90" containerName="registry" containerID="cri-o://2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed" gracePeriod=30
Sep 15 18:56:10.370098 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.370075    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:56:10.384362 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384306    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-trusted-ca\") pod \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") "
Sep 15 18:56:10.384469 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384424    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-certificates\") pod \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") "
Sep 15 18:56:10.384520 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384486    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-image-registry-private-configuration\") pod \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") "
Sep 15 18:56:10.384565 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384524    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-installation-pull-secrets\") pod \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") "
Sep 15 18:56:10.384565 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384550    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") pod \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") "
Sep 15 18:56:10.384689 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384591    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-bound-sa-token\") pod \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") "
Sep 15 18:56:10.384689 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384669    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e66c4f4a-c7e9-4a1f-9313-721feeee1c90" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:10.384689 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384681    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtzxt\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-kube-api-access-vtzxt\") pod \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") "
Sep 15 18:56:10.384843 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384723    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-ca-trust-extracted\") pod \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\" (UID: \"e66c4f4a-c7e9-4a1f-9313-721feeee1c90\") "
Sep 15 18:56:10.384843 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.384771    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e66c4f4a-c7e9-4a1f-9313-721feeee1c90" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:10.385053 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.385008    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-trusted-ca\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:56:10.385053 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.385029    2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-certificates\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:56:10.387466 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.387439    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e66c4f4a-c7e9-4a1f-9313-721feeee1c90" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:56:10.387575 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.387491    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-kube-api-access-vtzxt" (OuterVolumeSpecName: "kube-api-access-vtzxt") pod "e66c4f4a-c7e9-4a1f-9313-721feeee1c90" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90"). InnerVolumeSpecName "kube-api-access-vtzxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:56:10.387575 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.387525    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e66c4f4a-c7e9-4a1f-9313-721feeee1c90" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:56:10.387575 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.387544    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e66c4f4a-c7e9-4a1f-9313-721feeee1c90" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:56:10.387724 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.387632    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e66c4f4a-c7e9-4a1f-9313-721feeee1c90" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:56:10.395931 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.395906    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e66c4f4a-c7e9-4a1f-9313-721feeee1c90" (UID: "e66c4f4a-c7e9-4a1f-9313-721feeee1c90"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:56:10.485978 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.485946    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtzxt\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-kube-api-access-vtzxt\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:56:10.485978 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.485972    2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-ca-trust-extracted\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:56:10.485978 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.485984    2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-image-registry-private-configuration\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:56:10.486190 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.485993    2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-installation-pull-secrets\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:56:10.486190 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.486002    2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-registry-tls\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:56:10.486190 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.486011    2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e66c4f4a-c7e9-4a1f-9313-721feeee1c90-bound-sa-token\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:56:10.832826 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.832746    2556 generic.go:356] "Generic (PLEG): container finished" podID="e66c4f4a-c7e9-4a1f-9313-721feeee1c90" containerID="2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed" exitCode=0
Sep 15 18:56:10.832826 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.832790    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c555597d-46l4z"
Sep 15 18:56:10.832826 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.832819    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c555597d-46l4z" event={"ID":"e66c4f4a-c7e9-4a1f-9313-721feeee1c90","Type":"ContainerDied","Data":"2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed"}
Sep 15 18:56:10.833047 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.832855    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c555597d-46l4z" event={"ID":"e66c4f4a-c7e9-4a1f-9313-721feeee1c90","Type":"ContainerDied","Data":"e6ba396976d7efd8957c5333397b0978107322ffbe0af5de40825f4360fcd8e2"}
Sep 15 18:56:10.833047 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.832875    2556 scope.go:117] "RemoveContainer" containerID="2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed"
Sep 15 18:56:10.842079 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.842063    2556 scope.go:117] "RemoveContainer" containerID="2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed"
Sep 15 18:56:10.842339 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:56:10.842319    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed\": container with ID starting with 2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed not found: ID does not exist" containerID="2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed"
Sep 15 18:56:10.842408 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.842351    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed"} err="failed to get container status \"2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed\": rpc error: code = NotFound desc = could not find container \"2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed\": container with ID starting with 2a27ed7d390b9e28fa468807efcfe57e0b01d68ddfd819e069076f861c0c1fed not found: ID does not exist"
Sep 15 18:56:10.854653 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.854614    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76c555597d-46l4z"]
Sep 15 18:56:10.860256 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:10.860227    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76c555597d-46l4z"]
Sep 15 18:56:11.683014 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:11.678309    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e66c4f4a-c7e9-4a1f-9313-721feeee1c90" path="/var/lib/kubelet/pods/e66c4f4a-c7e9-4a1f-9313-721feeee1c90/volumes"
Sep 15 18:56:12.843600 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:12.843562    2556 generic.go:356] "Generic (PLEG): container finished" podID="27cc90e0-3da1-4615-96f3-fa7b0f89488b" containerID="c0db88ec658035478e69a660b804fec17b9aec9a40970605d2f1fd33074849c2" exitCode=0
Sep 15 18:56:12.844009 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:12.843633    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw" event={"ID":"27cc90e0-3da1-4615-96f3-fa7b0f89488b","Type":"ContainerDied","Data":"c0db88ec658035478e69a660b804fec17b9aec9a40970605d2f1fd33074849c2"}
Sep 15 18:56:12.844009 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:12.843946    2556 scope.go:117] "RemoveContainer" containerID="c0db88ec658035478e69a660b804fec17b9aec9a40970605d2f1fd33074849c2"
Sep 15 18:56:13.850060 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:13.850026    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jw4jw" event={"ID":"27cc90e0-3da1-4615-96f3-fa7b0f89488b","Type":"ContainerStarted","Data":"9b8e1467ee4513a6b5fac9c770e13b26cdb9a123d16013e045555b93f0c4cb86"}
Sep 15 18:56:20.533816 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:20.533785    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:56:20.537586 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:20.537558    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86f978c87-mbp62"
Sep 15 18:56:32.930872 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:32.930836    2556 generic.go:356] "Generic (PLEG): container finished" podID="f470421a-2714-4f71-9eb4-f2365287496a" containerID="9a7201afe5fabba4c29c7e3389a5814d5618f34d5bdfbaf8f47c1151ee77a856" exitCode=0
Sep 15 18:56:32.930872 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:32.930875    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-8n8xn" event={"ID":"f470421a-2714-4f71-9eb4-f2365287496a","Type":"ContainerDied","Data":"9a7201afe5fabba4c29c7e3389a5814d5618f34d5bdfbaf8f47c1151ee77a856"}
Sep 15 18:56:32.931328 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:32.931187    2556 scope.go:117] "RemoveContainer" containerID="9a7201afe5fabba4c29c7e3389a5814d5618f34d5bdfbaf8f47c1151ee77a856"
Sep 15 18:56:33.936900 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:33.936814    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-8n8xn" event={"ID":"f470421a-2714-4f71-9eb4-f2365287496a","Type":"ContainerStarted","Data":"fdcecfc6130140c42345f02dbdafe05de03297dcb11b7d4c1f0785ed91d06967"}
Sep 15 18:56:42.279442 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:42.279408    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:56:42.298100 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:42.298078    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:56:42.991121 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:42.991092    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:56:55.574130 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:55.574091    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:56:55.576498 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:55.576464    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0724acd2-fe1b-4134-817d-a4fc7300ce5c-metrics-certs\") pod \"network-metrics-daemon-hfjb8\" (UID: \"0724acd2-fe1b-4134-817d-a4fc7300ce5c\") " pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:56:55.675307 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:55.675275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bc8c\""
Sep 15 18:56:55.683428 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:55.683400    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfjb8"
Sep 15 18:56:55.953598 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:55.953561    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hfjb8"]
Sep 15 18:56:55.957224 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:56:55.957193    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0724acd2_fe1b_4134_817d_a4fc7300ce5c.slice/crio-d693054c1ad29b4c7b5cda66bba7422798bce40fca57f62b6b9e9f5115c79266 WatchSource:0}: Error finding container d693054c1ad29b4c7b5cda66bba7422798bce40fca57f62b6b9e9f5115c79266: Status 404 returned error can't find the container with id d693054c1ad29b4c7b5cda66bba7422798bce40fca57f62b6b9e9f5115c79266
Sep 15 18:56:56.036020 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:56.035992    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hfjb8" event={"ID":"0724acd2-fe1b-4134-817d-a4fc7300ce5c","Type":"ContainerStarted","Data":"d693054c1ad29b4c7b5cda66bba7422798bce40fca57f62b6b9e9f5115c79266"}
Sep 15 18:56:58.046267 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:58.046228    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hfjb8" event={"ID":"0724acd2-fe1b-4134-817d-a4fc7300ce5c","Type":"ContainerStarted","Data":"646b38c0a2f9a557eb3eca3fa72ed132567731e71f2685e4e0af2d3a5936901a"}
Sep 15 18:56:58.046613 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:58.046290    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hfjb8" event={"ID":"0724acd2-fe1b-4134-817d-a4fc7300ce5c","Type":"ContainerStarted","Data":"99a912fa81cd40e859167b2ecdd2c9ab1f8be8c24424cb82586ae750df7f8858"}
Sep 15 18:56:58.061823 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:56:58.061778    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hfjb8" podStartSLOduration=374.487642079 podStartE2EDuration="6m16.061762893s" totalImagesPullingTime="1.574120814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:56:58.060482028 +0000 UTC m=+376.916335297" watchObservedRunningTime="2026-09-15 18:56:58.061762893 +0000 UTC m=+376.917616160"
Sep 15 18:57:15.468401 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:15.468367    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-979cd9b44-jxqss"]
Sep 15 18:57:40.492166 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.492120    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-979cd9b44-jxqss" podUID="7e4dead5-80fc-4dae-9c5d-244313248689" containerName="console" containerID="cri-o://263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7" gracePeriod=15
Sep 15 18:57:40.738892 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.738864    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:57:40.827111 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827032    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-console-config\") pod \"7e4dead5-80fc-4dae-9c5d-244313248689\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") "
Sep 15 18:57:40.827111 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827072    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-oauth-config\") pod \"7e4dead5-80fc-4dae-9c5d-244313248689\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") "
Sep 15 18:57:40.827331 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827117    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-oauth-serving-cert\") pod \"7e4dead5-80fc-4dae-9c5d-244313248689\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") "
Sep 15 18:57:40.827331 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827220    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h47wk\" (UniqueName: \"kubernetes.io/projected/7e4dead5-80fc-4dae-9c5d-244313248689-kube-api-access-h47wk\") pod \"7e4dead5-80fc-4dae-9c5d-244313248689\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") "
Sep 15 18:57:40.827331 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827274    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-trusted-ca-bundle\") pod \"7e4dead5-80fc-4dae-9c5d-244313248689\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") "
Sep 15 18:57:40.827487 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827334    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-service-ca\") pod \"7e4dead5-80fc-4dae-9c5d-244313248689\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") "
Sep 15 18:57:40.827487 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827359    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-serving-cert\") pod \"7e4dead5-80fc-4dae-9c5d-244313248689\" (UID: \"7e4dead5-80fc-4dae-9c5d-244313248689\") "
Sep 15 18:57:40.827487 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827477    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-console-config" (OuterVolumeSpecName: "console-config") pod "7e4dead5-80fc-4dae-9c5d-244313248689" (UID: "7e4dead5-80fc-4dae-9c5d-244313248689"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:40.827635 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827595    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-console-config\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:57:40.827635 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827609    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7e4dead5-80fc-4dae-9c5d-244313248689" (UID: "7e4dead5-80fc-4dae-9c5d-244313248689"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:40.827883 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827850    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7e4dead5-80fc-4dae-9c5d-244313248689" (UID: "7e4dead5-80fc-4dae-9c5d-244313248689"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:40.828018 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.827942    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-service-ca" (OuterVolumeSpecName: "service-ca") pod "7e4dead5-80fc-4dae-9c5d-244313248689" (UID: "7e4dead5-80fc-4dae-9c5d-244313248689"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:40.829487 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.829463    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7e4dead5-80fc-4dae-9c5d-244313248689" (UID: "7e4dead5-80fc-4dae-9c5d-244313248689"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:57:40.829487 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.829476    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4dead5-80fc-4dae-9c5d-244313248689-kube-api-access-h47wk" (OuterVolumeSpecName: "kube-api-access-h47wk") pod "7e4dead5-80fc-4dae-9c5d-244313248689" (UID: "7e4dead5-80fc-4dae-9c5d-244313248689"). InnerVolumeSpecName "kube-api-access-h47wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:57:40.829610 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.829491    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7e4dead5-80fc-4dae-9c5d-244313248689" (UID: "7e4dead5-80fc-4dae-9c5d-244313248689"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:57:40.928809 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.928781    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-oauth-config\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:57:40.928809 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.928807    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-oauth-serving-cert\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:57:40.928976 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.928818    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47wk\" (UniqueName: \"kubernetes.io/projected/7e4dead5-80fc-4dae-9c5d-244313248689-kube-api-access-h47wk\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:57:40.928976 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.928827    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-trusted-ca-bundle\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:57:40.928976 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.928837    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4dead5-80fc-4dae-9c5d-244313248689-service-ca\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:57:40.928976 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:40.928847    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dead5-80fc-4dae-9c5d-244313248689-console-serving-cert\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 18:57:41.226470 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.226434    2556 generic.go:356] "Generic (PLEG): container finished" podID="7e4dead5-80fc-4dae-9c5d-244313248689" containerID="263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7" exitCode=0
Sep 15 18:57:41.226632 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.226478    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-979cd9b44-jxqss"
Sep 15 18:57:41.226632 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.226486    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-979cd9b44-jxqss" event={"ID":"7e4dead5-80fc-4dae-9c5d-244313248689","Type":"ContainerDied","Data":"263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7"}
Sep 15 18:57:41.226632 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.226529    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-979cd9b44-jxqss" event={"ID":"7e4dead5-80fc-4dae-9c5d-244313248689","Type":"ContainerDied","Data":"d9c7a766cac0de06ec658d75d8352a41e0b8fe565caa224c4f52f859db366594"}
Sep 15 18:57:41.226632 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.226549    2556 scope.go:117] "RemoveContainer" containerID="263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7"
Sep 15 18:57:41.235466 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.235446    2556 scope.go:117] "RemoveContainer" containerID="263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7"
Sep 15 18:57:41.235761 ip-10-0-140-183 kubenswrapper[2556]: E0915 18:57:41.235738    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7\": container with ID starting with 263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7 not found: ID does not exist" containerID="263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7"
Sep 15 18:57:41.235845 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.235765    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7"} err="failed to get container status \"263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7\": rpc error: code = NotFound desc = could not find container \"263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7\": container with ID starting with 263aca215e2c81e688b3d70187bbc38beb356beb1c307694f58436b7c403aff7 not found: ID does not exist"
Sep 15 18:57:41.247403 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.247376    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-979cd9b44-jxqss"]
Sep 15 18:57:41.248683 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.248660    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-979cd9b44-jxqss"]
Sep 15 18:57:41.683403 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:57:41.683372    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e4dead5-80fc-4dae-9c5d-244313248689" path="/var/lib/kubelet/pods/7e4dead5-80fc-4dae-9c5d-244313248689/volumes"
Sep 15 18:59:19.332978 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.332940    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bqcbs"]
Sep 15 18:59:19.333422 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.333261    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4dead5-80fc-4dae-9c5d-244313248689" containerName="console"
Sep 15 18:59:19.333422 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.333273    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4dead5-80fc-4dae-9c5d-244313248689" containerName="console"
Sep 15 18:59:19.333422 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.333291    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66c4f4a-c7e9-4a1f-9313-721feeee1c90" containerName="registry"
Sep 15 18:59:19.333422 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.333296    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66c4f4a-c7e9-4a1f-9313-721feeee1c90" containerName="registry"
Sep 15 18:59:19.333422 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.333344    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e4dead5-80fc-4dae-9c5d-244313248689" containerName="console"
Sep 15 18:59:19.333422 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.333359    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66c4f4a-c7e9-4a1f-9313-721feeee1c90" containerName="registry"
Sep 15 18:59:19.336483 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.336466    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bqcbs"
Sep 15 18:59:19.338892 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.338858    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 18:59:19.338892 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.338883    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 18:59:19.339767 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.339748    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nlbvw\""
Sep 15 18:59:19.345477 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.345455    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bqcbs"]
Sep 15 18:59:19.414714 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.414689    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3114c55c-f15a-4e8e-86b5-964ee7ab14e5-bound-sa-token\") pod \"cert-manager-55b4d954d5-bqcbs\" (UID: \"3114c55c-f15a-4e8e-86b5-964ee7ab14e5\") " pod="cert-manager/cert-manager-55b4d954d5-bqcbs"
Sep 15 18:59:19.414828 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.414732    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczh9\" (UniqueName: \"kubernetes.io/projected/3114c55c-f15a-4e8e-86b5-964ee7ab14e5-kube-api-access-dczh9\") pod \"cert-manager-55b4d954d5-bqcbs\" (UID: \"3114c55c-f15a-4e8e-86b5-964ee7ab14e5\") " pod="cert-manager/cert-manager-55b4d954d5-bqcbs"
Sep 15 18:59:19.515407 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.515375    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczh9\" (UniqueName: \"kubernetes.io/projected/3114c55c-f15a-4e8e-86b5-964ee7ab14e5-kube-api-access-dczh9\") pod \"cert-manager-55b4d954d5-bqcbs\" (UID: \"3114c55c-f15a-4e8e-86b5-964ee7ab14e5\") " pod="cert-manager/cert-manager-55b4d954d5-bqcbs"
Sep 15 18:59:19.515553 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.515473    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3114c55c-f15a-4e8e-86b5-964ee7ab14e5-bound-sa-token\") pod \"cert-manager-55b4d954d5-bqcbs\" (UID: \"3114c55c-f15a-4e8e-86b5-964ee7ab14e5\") " pod="cert-manager/cert-manager-55b4d954d5-bqcbs"
Sep 15 18:59:19.523027 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.523004    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3114c55c-f15a-4e8e-86b5-964ee7ab14e5-bound-sa-token\") pod \"cert-manager-55b4d954d5-bqcbs\" (UID: \"3114c55c-f15a-4e8e-86b5-964ee7ab14e5\") " pod="cert-manager/cert-manager-55b4d954d5-bqcbs"
Sep 15 18:59:19.523116 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.523077    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczh9\" (UniqueName: \"kubernetes.io/projected/3114c55c-f15a-4e8e-86b5-964ee7ab14e5-kube-api-access-dczh9\") pod \"cert-manager-55b4d954d5-bqcbs\" (UID: \"3114c55c-f15a-4e8e-86b5-964ee7ab14e5\") " pod="cert-manager/cert-manager-55b4d954d5-bqcbs"
Sep 15 18:59:19.652541 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.652512    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bqcbs"
Sep 15 18:59:19.921678 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:19.921651    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bqcbs"]
Sep 15 18:59:19.923811 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:59:19.923778    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3114c55c_f15a_4e8e_86b5_964ee7ab14e5.slice/crio-dec077e001a0ee5d329f3a81059157d47db35146b653fae1c20e9d47561e93e6 WatchSource:0}: Error finding container dec077e001a0ee5d329f3a81059157d47db35146b653fae1c20e9d47561e93e6: Status 404 returned error can't find the container with id dec077e001a0ee5d329f3a81059157d47db35146b653fae1c20e9d47561e93e6
Sep 15 18:59:20.627801 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:20.627766    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bqcbs" event={"ID":"3114c55c-f15a-4e8e-86b5-964ee7ab14e5","Type":"ContainerStarted","Data":"dec077e001a0ee5d329f3a81059157d47db35146b653fae1c20e9d47561e93e6"}
Sep 15 18:59:25.652285 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:25.652240    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bqcbs" event={"ID":"3114c55c-f15a-4e8e-86b5-964ee7ab14e5","Type":"ContainerStarted","Data":"32019d7345201485f65923b3cf1fdbf4a357b9a3bf3a1c0347b9653131f05a36"}
Sep 15 18:59:25.666201 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:25.666156    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bqcbs" podStartSLOduration=1.418647468 podStartE2EDuration="6.666141752s" totalImagesPullingTime="5.247494284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:25.665403175 +0000 UTC m=+524.521256443" watchObservedRunningTime="2026-09-15 18:59:25.666141752 +0000 UTC m=+524.521995020"
Sep 15 18:59:28.566549 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.566515    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"]
Sep 15 18:59:28.569990 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.569971    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.572701 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.572681    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 18:59:28.573743 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.573725    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:59:28.574037 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.574018    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 18:59:28.574123 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.574039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 18:59:28.574123 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.574075    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 18:59:28.574213 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.574157    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dbcbm\""
Sep 15 18:59:28.580459 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.580437    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"]
Sep 15 18:59:28.689805 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.689780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d25567c7-4a0f-4048-bee6-7f8bc83a599b-metrics-cert\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.689956 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.689812    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25567c7-4a0f-4048-bee6-7f8bc83a599b-cert\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.689956 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.689858    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d25567c7-4a0f-4048-bee6-7f8bc83a599b-manager-config\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.689956 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.689874    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjgd\" (UniqueName: \"kubernetes.io/projected/d25567c7-4a0f-4048-bee6-7f8bc83a599b-kube-api-access-7kjgd\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.791031 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.790989    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d25567c7-4a0f-4048-bee6-7f8bc83a599b-manager-config\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.791031 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.791028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjgd\" (UniqueName: \"kubernetes.io/projected/d25567c7-4a0f-4048-bee6-7f8bc83a599b-kube-api-access-7kjgd\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.791235 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.791088    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d25567c7-4a0f-4048-bee6-7f8bc83a599b-metrics-cert\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.791235 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.791127    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25567c7-4a0f-4048-bee6-7f8bc83a599b-cert\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.791598 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.791579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d25567c7-4a0f-4048-bee6-7f8bc83a599b-manager-config\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.793521 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.793500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25567c7-4a0f-4048-bee6-7f8bc83a599b-cert\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.793583 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.793555    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d25567c7-4a0f-4048-bee6-7f8bc83a599b-metrics-cert\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.799224 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.799207    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjgd\" (UniqueName: \"kubernetes.io/projected/d25567c7-4a0f-4048-bee6-7f8bc83a599b-kube-api-access-7kjgd\") pod \"lws-controller-manager-5bcb5dcf99-6thb8\" (UID: \"d25567c7-4a0f-4048-bee6-7f8bc83a599b\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:28.879907 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:28.879874    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:29.152374 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:29.152350    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"]
Sep 15 18:59:29.154703 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:59:29.154676    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25567c7_4a0f_4048_bee6_7f8bc83a599b.slice/crio-1d96efe2e1cae5371287c8f1cb510d6e22c4c37ef7284f6e5d861a2db6c9a96d WatchSource:0}: Error finding container 1d96efe2e1cae5371287c8f1cb510d6e22c4c37ef7284f6e5d861a2db6c9a96d: Status 404 returned error can't find the container with id 1d96efe2e1cae5371287c8f1cb510d6e22c4c37ef7284f6e5d861a2db6c9a96d
Sep 15 18:59:29.670498 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:29.670463    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8" event={"ID":"d25567c7-4a0f-4048-bee6-7f8bc83a599b","Type":"ContainerStarted","Data":"1d96efe2e1cae5371287c8f1cb510d6e22c4c37ef7284f6e5d861a2db6c9a96d"}
Sep 15 18:59:32.686368 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:32.686336    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8" event={"ID":"d25567c7-4a0f-4048-bee6-7f8bc83a599b","Type":"ContainerStarted","Data":"f9b4dae418890adffaf456a355484267f227673276d778d3f127fe1683538256"}
Sep 15 18:59:32.686849 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:32.686419    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:32.702020 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:32.701955    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8" podStartSLOduration=2.20176995 podStartE2EDuration="4.701942313s" totalImagesPullingTime="2.500172363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:32.701461375 +0000 UTC m=+531.557314659" watchObservedRunningTime="2026-09-15 18:59:32.701942313 +0000 UTC m=+531.557795579"
Sep 15 18:59:38.676986 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.676906    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"]
Sep 15 18:59:38.680362 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.680345    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.682791 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.682768    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 18:59:38.682791 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.682766    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 18:59:38.682966 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.682810    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 18:59:38.682966 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.682863    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 18:59:38.683048 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.683013    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wdkzs\""
Sep 15 18:59:38.696531 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.696503    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"]
Sep 15 18:59:38.782222 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.782193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdjf\" (UniqueName: \"kubernetes.io/projected/21361244-e888-41b9-8673-c3c5d86f9e39-kube-api-access-hhdjf\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.782347 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.782229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21361244-e888-41b9-8673-c3c5d86f9e39-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.782347 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.782305    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21361244-e888-41b9-8673-c3c5d86f9e39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.782417 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.782375    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21361244-e888-41b9-8673-c3c5d86f9e39-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.782454 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.782431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21361244-e888-41b9-8673-c3c5d86f9e39-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.882922 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.882897    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21361244-e888-41b9-8673-c3c5d86f9e39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.883020 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.882931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21361244-e888-41b9-8673-c3c5d86f9e39-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.883020 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.882975    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21361244-e888-41b9-8673-c3c5d86f9e39-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.883020 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.883006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdjf\" (UniqueName: \"kubernetes.io/projected/21361244-e888-41b9-8673-c3c5d86f9e39-kube-api-access-hhdjf\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.883128 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.883028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21361244-e888-41b9-8673-c3c5d86f9e39-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.883314 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.883296    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21361244-e888-41b9-8673-c3c5d86f9e39-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.885363 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.885331    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21361244-e888-41b9-8673-c3c5d86f9e39-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.885547 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.885527    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21361244-e888-41b9-8673-c3c5d86f9e39-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.885620 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.885551    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21361244-e888-41b9-8673-c3c5d86f9e39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.895065 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.895046    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdjf\" (UniqueName: \"kubernetes.io/projected/21361244-e888-41b9-8673-c3c5d86f9e39-kube-api-access-hhdjf\") pod \"opendatahub-operator-controller-manager-6d9df7495-mqjpb\" (UID: \"21361244-e888-41b9-8673-c3c5d86f9e39\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:38.990383 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:38.990312    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:39.264337 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:39.264271    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"]
Sep 15 18:59:39.266965 ip-10-0-140-183 kubenswrapper[2556]: W0915 18:59:39.266937    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21361244_e888_41b9_8673_c3c5d86f9e39.slice/crio-7f80c160f1e87cde0e0140a88a8708da29a57afca4a2b4cbd84da4da81a30057 WatchSource:0}: Error finding container 7f80c160f1e87cde0e0140a88a8708da29a57afca4a2b4cbd84da4da81a30057: Status 404 returned error can't find the container with id 7f80c160f1e87cde0e0140a88a8708da29a57afca4a2b4cbd84da4da81a30057
Sep 15 18:59:39.716038 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:39.716002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb" event={"ID":"21361244-e888-41b9-8673-c3c5d86f9e39","Type":"ContainerStarted","Data":"7f80c160f1e87cde0e0140a88a8708da29a57afca4a2b4cbd84da4da81a30057"}
Sep 15 18:59:42.733570 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:42.733533    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb" event={"ID":"21361244-e888-41b9-8673-c3c5d86f9e39","Type":"ContainerStarted","Data":"c8f43fc624940e92bf8a197c7e28774c0e47668c8fd994ed73b9f667e65689b7"}
Sep 15 18:59:43.693276 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:43.693249    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-6thb8"
Sep 15 18:59:43.740242 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:43.740204    2556 generic.go:356] "Generic (PLEG): container finished" podID="21361244-e888-41b9-8673-c3c5d86f9e39" containerID="c8f43fc624940e92bf8a197c7e28774c0e47668c8fd994ed73b9f667e65689b7" exitCode=0
Sep 15 18:59:43.740725 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:43.740300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb" event={"ID":"21361244-e888-41b9-8673-c3c5d86f9e39","Type":"ContainerDied","Data":"c8f43fc624940e92bf8a197c7e28774c0e47668c8fd994ed73b9f667e65689b7"}
Sep 15 18:59:44.748743 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:44.748701    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb" event={"ID":"21361244-e888-41b9-8673-c3c5d86f9e39","Type":"ContainerStarted","Data":"ab36014077627773119e7ed0ae5adbb6b4721bbc49b4c64a2a9e2627aacdb037"}
Sep 15 18:59:44.749237 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:44.748822    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 18:59:44.771247 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:44.771196    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb" podStartSLOduration=2.445813521 podStartE2EDuration="6.771180948s" totalImagesPullingTime="3.281219416s" totalInitContainerRuntime="1.044148011s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:44.769072114 +0000 UTC m=+543.624925377" watchObservedRunningTime="2026-09-15 18:59:44.771180948 +0000 UTC m=+543.627034216"
Sep 15 18:59:55.755820 ip-10-0-140-183 kubenswrapper[2556]: I0915 18:59:55.755790    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-mqjpb"
Sep 15 19:00:10.445181 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.445141    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"]
Sep 15 19:00:10.449220 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.449198    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:10.451791 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.451511    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 19:00:10.451791 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.451545    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 19:00:10.451791 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.451618    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 19:00:10.451791 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.451620    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 19:00:10.452523 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.452502    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fmg74\""
Sep 15 19:00:10.458706 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.458685    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"]
Sep 15 19:00:10.553504 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.553473    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tmp\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:10.553504 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.553509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tls-certs\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:10.553765 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.553534    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjlcr\" (UniqueName: \"kubernetes.io/projected/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-kube-api-access-sjlcr\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:10.655073 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.655036    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tmp\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:10.655073 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.655077    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tls-certs\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:10.655267 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.655102    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjlcr\" (UniqueName: \"kubernetes.io/projected/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-kube-api-access-sjlcr\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:10.655267 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:00:10.655200    2556 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 15 19:00:10.655333 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:00:10.655287    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tls-certs podName:42e7b5de-d52b-45b5-b510-f4bae1fd4fc8 nodeName:}" failed. No retries permitted until 2026-09-15 19:00:11.155265982 +0000 UTC m=+570.011119231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tls-certs") pod "kube-auth-proxy-65488ff6c7-8b8lm" (UID: "42e7b5de-d52b-45b5-b510-f4bae1fd4fc8") : secret "kube-auth-proxy-tls" not found
Sep 15 19:00:10.657360 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.657339    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tmp\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:10.663031 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:10.662986    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjlcr\" (UniqueName: \"kubernetes.io/projected/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-kube-api-access-sjlcr\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:11.158691 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:11.158625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tls-certs\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:11.161308 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:11.161281    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7b5de-d52b-45b5-b510-f4bae1fd4fc8-tls-certs\") pod \"kube-auth-proxy-65488ff6c7-8b8lm\" (UID: \"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:11.361832 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:11.361784    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"
Sep 15 19:00:11.636397 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:11.636374    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm"]
Sep 15 19:00:11.638635 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:00:11.638610    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e7b5de_d52b_45b5_b510_f4bae1fd4fc8.slice/crio-887e3d8d1d81327be9e368985e1301f52990a646d0982869a70cc5c07964adc8 WatchSource:0}: Error finding container 887e3d8d1d81327be9e368985e1301f52990a646d0982869a70cc5c07964adc8: Status 404 returned error can't find the container with id 887e3d8d1d81327be9e368985e1301f52990a646d0982869a70cc5c07964adc8
Sep 15 19:00:11.877002 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:11.876964    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm" event={"ID":"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8","Type":"ContainerStarted","Data":"887e3d8d1d81327be9e368985e1301f52990a646d0982869a70cc5c07964adc8"}
Sep 15 19:00:15.901517 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:15.901473    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm" event={"ID":"42e7b5de-d52b-45b5-b510-f4bae1fd4fc8","Type":"ContainerStarted","Data":"a5f955efe83378c1117c6a1a5801bacd7d3e220593953de6341d910d564e781f"}
Sep 15 19:00:15.918419 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:15.918366    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-8b8lm" podStartSLOduration=2.56859448 podStartE2EDuration="5.918352428s" totalImagesPullingTime="3.349757948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:15.916174941 +0000 UTC m=+574.772028209" watchObservedRunningTime="2026-09-15 19:00:15.918352428 +0000 UTC m=+574.774205696"
Sep 15 19:00:41.667550 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:41.667435    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:00:41.671656 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:41.668753    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:00:41.672550 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:41.672528    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:00:41.673998 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:00:41.673972    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:02:42.473293 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.473213    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mmscc"]
Sep 15 19:02:42.476616 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.476594    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:42.478848 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.478825    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jsd5d\""
Sep 15 19:02:42.478963 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.478825    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 19:02:42.478963 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.478896    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 19:02:42.479601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.479584    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 19:02:42.483955 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.483935    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mmscc"]
Sep 15 19:02:42.574788 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.574714    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mmscc"]
Sep 15 19:02:42.576307 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.576271    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxwb\" (UniqueName: \"kubernetes.io/projected/9e8a6540-78c5-43fb-9b75-dabecc376111-kube-api-access-scxwb\") pod \"limitador-limitador-8588749977-mmscc\" (UID: \"9e8a6540-78c5-43fb-9b75-dabecc376111\") " pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:42.576462 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.576391    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e8a6540-78c5-43fb-9b75-dabecc376111-config-file\") pod \"limitador-limitador-8588749977-mmscc\" (UID: \"9e8a6540-78c5-43fb-9b75-dabecc376111\") " pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:42.677115 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.677085    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxwb\" (UniqueName: \"kubernetes.io/projected/9e8a6540-78c5-43fb-9b75-dabecc376111-kube-api-access-scxwb\") pod \"limitador-limitador-8588749977-mmscc\" (UID: \"9e8a6540-78c5-43fb-9b75-dabecc376111\") " pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:42.677263 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.677141    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e8a6540-78c5-43fb-9b75-dabecc376111-config-file\") pod \"limitador-limitador-8588749977-mmscc\" (UID: \"9e8a6540-78c5-43fb-9b75-dabecc376111\") " pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:42.677733 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.677715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e8a6540-78c5-43fb-9b75-dabecc376111-config-file\") pod \"limitador-limitador-8588749977-mmscc\" (UID: \"9e8a6540-78c5-43fb-9b75-dabecc376111\") " pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:42.684652 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.684624    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxwb\" (UniqueName: \"kubernetes.io/projected/9e8a6540-78c5-43fb-9b75-dabecc376111-kube-api-access-scxwb\") pod \"limitador-limitador-8588749977-mmscc\" (UID: \"9e8a6540-78c5-43fb-9b75-dabecc376111\") " pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:42.787268 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:42.787185    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:43.058587 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.058563    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mmscc"]
Sep 15 19:02:43.061138 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:02:43.061109    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8a6540_78c5_43fb_9b75_dabecc376111.slice/crio-1713fda3caf107bde692f867ca372872de3a6ed411712806ef15df5963e17ed3 WatchSource:0}: Error finding container 1713fda3caf107bde692f867ca372872de3a6ed411712806ef15df5963e17ed3: Status 404 returned error can't find the container with id 1713fda3caf107bde692f867ca372872de3a6ed411712806ef15df5963e17ed3
Sep 15 19:02:43.063185 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.063170    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:02:43.375748 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.375716    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-q2prw"]
Sep 15 19:02:43.380889 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.380863    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-q2prw"
Sep 15 19:02:43.383138 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.383112    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mzzjb\""
Sep 15 19:02:43.385586 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.385555    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q2prw"]
Sep 15 19:02:43.484447 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.484415    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czx4\" (UniqueName: \"kubernetes.io/projected/15cd73ae-05c6-4d24-a688-ceebeb671f70-kube-api-access-6czx4\") pod \"authorino-59c974bd48-q2prw\" (UID: \"15cd73ae-05c6-4d24-a688-ceebeb671f70\") " pod="kuadrant-system/authorino-59c974bd48-q2prw"
Sep 15 19:02:43.585076 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.585044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czx4\" (UniqueName: \"kubernetes.io/projected/15cd73ae-05c6-4d24-a688-ceebeb671f70-kube-api-access-6czx4\") pod \"authorino-59c974bd48-q2prw\" (UID: \"15cd73ae-05c6-4d24-a688-ceebeb671f70\") " pod="kuadrant-system/authorino-59c974bd48-q2prw"
Sep 15 19:02:43.590101 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.590077    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mmscc" event={"ID":"9e8a6540-78c5-43fb-9b75-dabecc376111","Type":"ContainerStarted","Data":"1713fda3caf107bde692f867ca372872de3a6ed411712806ef15df5963e17ed3"}
Sep 15 19:02:43.592633 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.592609    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czx4\" (UniqueName: \"kubernetes.io/projected/15cd73ae-05c6-4d24-a688-ceebeb671f70-kube-api-access-6czx4\") pod \"authorino-59c974bd48-q2prw\" (UID: \"15cd73ae-05c6-4d24-a688-ceebeb671f70\") " pod="kuadrant-system/authorino-59c974bd48-q2prw"
Sep 15 19:02:43.692195 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.692118    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-q2prw"
Sep 15 19:02:43.958475 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:43.958451    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q2prw"]
Sep 15 19:02:43.960510 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:02:43.960481    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15cd73ae_05c6_4d24_a688_ceebeb671f70.slice/crio-002094e2df657efdc0f8604cb4b1ab97cd3b0fe9f44482c25b6bcaa1dde6765c WatchSource:0}: Error finding container 002094e2df657efdc0f8604cb4b1ab97cd3b0fe9f44482c25b6bcaa1dde6765c: Status 404 returned error can't find the container with id 002094e2df657efdc0f8604cb4b1ab97cd3b0fe9f44482c25b6bcaa1dde6765c
Sep 15 19:02:44.596771 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:44.596733    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-q2prw" event={"ID":"15cd73ae-05c6-4d24-a688-ceebeb671f70","Type":"ContainerStarted","Data":"002094e2df657efdc0f8604cb4b1ab97cd3b0fe9f44482c25b6bcaa1dde6765c"}
Sep 15 19:02:47.614326 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:47.614286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mmscc" event={"ID":"9e8a6540-78c5-43fb-9b75-dabecc376111","Type":"ContainerStarted","Data":"3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e"}
Sep 15 19:02:47.614788 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:47.614378    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:47.615817 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:47.615792    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-q2prw" event={"ID":"15cd73ae-05c6-4d24-a688-ceebeb671f70","Type":"ContainerStarted","Data":"f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea"}
Sep 15 19:02:47.630248 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:47.630178    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-mmscc" podStartSLOduration=1.981606356 podStartE2EDuration="5.630164616s" totalImagesPullingTime="3.64855826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:47.629235365 +0000 UTC m=+726.485088633" watchObservedRunningTime="2026-09-15 19:02:47.630164616 +0000 UTC m=+726.486017942"
Sep 15 19:02:47.642743 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:47.642704    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-q2prw" podStartSLOduration=1.8874465489999999 podStartE2EDuration="4.642690315s" totalImagesPullingTime="2.755243766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:47.642677736 +0000 UTC m=+726.498531006" watchObservedRunningTime="2026-09-15 19:02:47.642690315 +0000 UTC m=+726.498543576"
Sep 15 19:02:56.706353 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:56.706313    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mmscc"]
Sep 15 19:02:56.706841 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:56.706609    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-mmscc" podUID="9e8a6540-78c5-43fb-9b75-dabecc376111" containerName="limitador" containerID="cri-o://3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e" gracePeriod=30
Sep 15 19:02:56.707204 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:56.707170    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:57.253895 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.253874    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:57.411891 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.411865    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scxwb\" (UniqueName: \"kubernetes.io/projected/9e8a6540-78c5-43fb-9b75-dabecc376111-kube-api-access-scxwb\") pod \"9e8a6540-78c5-43fb-9b75-dabecc376111\" (UID: \"9e8a6540-78c5-43fb-9b75-dabecc376111\") "
Sep 15 19:02:57.412043 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.411906    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e8a6540-78c5-43fb-9b75-dabecc376111-config-file\") pod \"9e8a6540-78c5-43fb-9b75-dabecc376111\" (UID: \"9e8a6540-78c5-43fb-9b75-dabecc376111\") "
Sep 15 19:02:57.412292 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.412267    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8a6540-78c5-43fb-9b75-dabecc376111-config-file" (OuterVolumeSpecName: "config-file") pod "9e8a6540-78c5-43fb-9b75-dabecc376111" (UID: "9e8a6540-78c5-43fb-9b75-dabecc376111"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:02:57.413959 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.413940    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8a6540-78c5-43fb-9b75-dabecc376111-kube-api-access-scxwb" (OuterVolumeSpecName: "kube-api-access-scxwb") pod "9e8a6540-78c5-43fb-9b75-dabecc376111" (UID: "9e8a6540-78c5-43fb-9b75-dabecc376111"). InnerVolumeSpecName "kube-api-access-scxwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:02:57.512584 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.512561    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxwb\" (UniqueName: \"kubernetes.io/projected/9e8a6540-78c5-43fb-9b75-dabecc376111-kube-api-access-scxwb\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:02:57.512584 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.512583    2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e8a6540-78c5-43fb-9b75-dabecc376111-config-file\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:02:57.664272 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.664203    2556 generic.go:356] "Generic (PLEG): container finished" podID="9e8a6540-78c5-43fb-9b75-dabecc376111" containerID="3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e" exitCode=0
Sep 15 19:02:57.664272 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.664247    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mmscc"
Sep 15 19:02:57.664440 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.664288    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mmscc" event={"ID":"9e8a6540-78c5-43fb-9b75-dabecc376111","Type":"ContainerDied","Data":"3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e"}
Sep 15 19:02:57.664440 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.664339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mmscc" event={"ID":"9e8a6540-78c5-43fb-9b75-dabecc376111","Type":"ContainerDied","Data":"1713fda3caf107bde692f867ca372872de3a6ed411712806ef15df5963e17ed3"}
Sep 15 19:02:57.664440 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.664365    2556 scope.go:117] "RemoveContainer" containerID="3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e"
Sep 15 19:02:57.673058 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.673039    2556 scope.go:117] "RemoveContainer" containerID="3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e"
Sep 15 19:02:57.673340 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:02:57.673321    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e\": container with ID starting with 3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e not found: ID does not exist" containerID="3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e"
Sep 15 19:02:57.673387 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.673350    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e"} err="failed to get container status \"3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e\": rpc error: code = NotFound desc = could not find container \"3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e\": container with ID starting with 3fedc86281a4e46b8ebc20edd7c8f5a2f068a919856e6742a0cc819e7c5b046e not found: ID does not exist"
Sep 15 19:02:57.685082 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.685045    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mmscc"]
Sep 15 19:02:57.688874 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:57.688852    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mmscc"]
Sep 15 19:02:59.684692 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:02:59.684658    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e8a6540-78c5-43fb-9b75-dabecc376111" path="/var/lib/kubelet/pods/9e8a6540-78c5-43fb-9b75-dabecc376111/volumes"
Sep 15 19:03:01.852361 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.852324    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"]
Sep 15 19:03:01.852904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.852883    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8a6540-78c5-43fb-9b75-dabecc376111" containerName="limitador"
Sep 15 19:03:01.852994 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.852908    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8a6540-78c5-43fb-9b75-dabecc376111" containerName="limitador"
Sep 15 19:03:01.853046 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.853015    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8a6540-78c5-43fb-9b75-dabecc376111" containerName="limitador"
Sep 15 19:03:01.857787 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.857765    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:01.860201 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.860179    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 19:03:01.860307 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.860202    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2rz8j\""
Sep 15 19:03:01.861152 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.861131    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 15 19:03:01.861257 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.861155    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:03:01.865935 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.865913    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"]
Sep 15 19:03:01.953564 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.953520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbhn\" (UniqueName: \"kubernetes.io/projected/ba4f2de2-70f5-4f3e-923e-82ee297e0c97-kube-api-access-bxbhn\") pod \"postgres-69fb4594cf-mnqnv\" (UID: \"ba4f2de2-70f5-4f3e-923e-82ee297e0c97\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:01.953564 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:01.953574    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba4f2de2-70f5-4f3e-923e-82ee297e0c97-data\") pod \"postgres-69fb4594cf-mnqnv\" (UID: \"ba4f2de2-70f5-4f3e-923e-82ee297e0c97\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:02.054448 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:02.054411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbhn\" (UniqueName: \"kubernetes.io/projected/ba4f2de2-70f5-4f3e-923e-82ee297e0c97-kube-api-access-bxbhn\") pod \"postgres-69fb4594cf-mnqnv\" (UID: \"ba4f2de2-70f5-4f3e-923e-82ee297e0c97\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:02.054686 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:02.054471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba4f2de2-70f5-4f3e-923e-82ee297e0c97-data\") pod \"postgres-69fb4594cf-mnqnv\" (UID: \"ba4f2de2-70f5-4f3e-923e-82ee297e0c97\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:02.054923 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:02.054903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba4f2de2-70f5-4f3e-923e-82ee297e0c97-data\") pod \"postgres-69fb4594cf-mnqnv\" (UID: \"ba4f2de2-70f5-4f3e-923e-82ee297e0c97\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:02.062221 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:02.062192    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbhn\" (UniqueName: \"kubernetes.io/projected/ba4f2de2-70f5-4f3e-923e-82ee297e0c97-kube-api-access-bxbhn\") pod \"postgres-69fb4594cf-mnqnv\" (UID: \"ba4f2de2-70f5-4f3e-923e-82ee297e0c97\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:02.169361 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:02.169326    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:02.438493 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:02.438470    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"]
Sep 15 19:03:02.440503 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:03:02.440462    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4f2de2_70f5_4f3e_923e_82ee297e0c97.slice/crio-4dbfd8aa015d298c1ce37a943ea74f2063ff4bf06ed54d10b95e883973006eaf WatchSource:0}: Error finding container 4dbfd8aa015d298c1ce37a943ea74f2063ff4bf06ed54d10b95e883973006eaf: Status 404 returned error can't find the container with id 4dbfd8aa015d298c1ce37a943ea74f2063ff4bf06ed54d10b95e883973006eaf
Sep 15 19:03:02.695338 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:02.695250    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv" event={"ID":"ba4f2de2-70f5-4f3e-923e-82ee297e0c97","Type":"ContainerStarted","Data":"4dbfd8aa015d298c1ce37a943ea74f2063ff4bf06ed54d10b95e883973006eaf"}
Sep 15 19:03:07.727152 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:07.727123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv" event={"ID":"ba4f2de2-70f5-4f3e-923e-82ee297e0c97","Type":"ContainerStarted","Data":"72abf39f4c9dca3cef66305d0a763a6e18fa8a4e6120aaf08a23217786141f77"}
Sep 15 19:03:07.727528 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:07.727187    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:07.743565 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:07.743521    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv" podStartSLOduration=1.52712379 podStartE2EDuration="6.743504969s" totalImagesPullingTime="5.216381179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:07.742111825 +0000 UTC m=+746.597965094" watchObservedRunningTime="2026-09-15 19:03:07.743504969 +0000 UTC m=+746.599358237"
Sep 15 19:03:13.761480 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:13.761449    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mnqnv"
Sep 15 19:03:16.423389 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.423351    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q2prw"]
Sep 15 19:03:16.423789 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.423562    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-q2prw" podUID="15cd73ae-05c6-4d24-a688-ceebeb671f70" containerName="authorino" containerID="cri-o://f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea" gracePeriod=30
Sep 15 19:03:16.671831 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.671809    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-q2prw"
Sep 15 19:03:16.680791 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.680737    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6czx4\" (UniqueName: \"kubernetes.io/projected/15cd73ae-05c6-4d24-a688-ceebeb671f70-kube-api-access-6czx4\") pod \"15cd73ae-05c6-4d24-a688-ceebeb671f70\" (UID: \"15cd73ae-05c6-4d24-a688-ceebeb671f70\") "
Sep 15 19:03:16.682681 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.682628    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cd73ae-05c6-4d24-a688-ceebeb671f70-kube-api-access-6czx4" (OuterVolumeSpecName: "kube-api-access-6czx4") pod "15cd73ae-05c6-4d24-a688-ceebeb671f70" (UID: "15cd73ae-05c6-4d24-a688-ceebeb671f70"). InnerVolumeSpecName "kube-api-access-6czx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:03:16.771204 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.771170    2556 generic.go:356] "Generic (PLEG): container finished" podID="15cd73ae-05c6-4d24-a688-ceebeb671f70" containerID="f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea" exitCode=0
Sep 15 19:03:16.771310 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.771215    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-q2prw"
Sep 15 19:03:16.771310 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.771254    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-q2prw" event={"ID":"15cd73ae-05c6-4d24-a688-ceebeb671f70","Type":"ContainerDied","Data":"f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea"}
Sep 15 19:03:16.771310 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.771296    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-q2prw" event={"ID":"15cd73ae-05c6-4d24-a688-ceebeb671f70","Type":"ContainerDied","Data":"002094e2df657efdc0f8604cb4b1ab97cd3b0fe9f44482c25b6bcaa1dde6765c"}
Sep 15 19:03:16.771421 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.771313    2556 scope.go:117] "RemoveContainer" containerID="f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea"
Sep 15 19:03:16.781799 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.781775    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czx4\" (UniqueName: \"kubernetes.io/projected/15cd73ae-05c6-4d24-a688-ceebeb671f70-kube-api-access-6czx4\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:03:16.789503 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.789484    2556 scope.go:117] "RemoveContainer" containerID="f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea"
Sep 15 19:03:16.789803 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:03:16.789782    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea\": container with ID starting with f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea not found: ID does not exist" containerID="f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea"
Sep 15 19:03:16.789877 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.789810    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea"} err="failed to get container status \"f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea\": rpc error: code = NotFound desc = could not find container \"f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea\": container with ID starting with f63770b5fd936b4124fd704a18a03ac9d06372dcf7860b16be73e6378e6638ea not found: ID does not exist"
Sep 15 19:03:16.795571 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.795548    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q2prw"]
Sep 15 19:03:16.797885 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:16.797867    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q2prw"]
Sep 15 19:03:17.678125 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:17.678092    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15cd73ae-05c6-4d24-a688-ceebeb671f70" path="/var/lib/kubelet/pods/15cd73ae-05c6-4d24-a688-ceebeb671f70/volumes"
Sep 15 19:03:20.969922 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:20.969894    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-9mjmn"]
Sep 15 19:03:20.970300 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:20.970254    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15cd73ae-05c6-4d24-a688-ceebeb671f70" containerName="authorino"
Sep 15 19:03:20.970300 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:20.970264    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="15cd73ae-05c6-4d24-a688-ceebeb671f70" containerName="authorino"
Sep 15 19:03:20.970377 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:20.970328    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="15cd73ae-05c6-4d24-a688-ceebeb671f70" containerName="authorino"
Sep 15 19:03:21.149690 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.149660    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-9mjmn"]
Sep 15 19:03:21.149841 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.149703    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-ndqln"]
Sep 15 19:03:21.149880 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.149840    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.152060 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.152035    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 19:03:21.152966 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.152946    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 19:03:21.153102 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.153063    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vc5j9\""
Sep 15 19:03:21.153177 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.153113    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 19:03:21.169330 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.169312    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-ndqln"]
Sep 15 19:03:21.169425 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.169393    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.217792 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.217770    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.217904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.217817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-webhook-cert\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.217904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.217849    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.217904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.217867    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296hj\" (UniqueName: \"kubernetes.io/projected/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-kube-api-access-296hj\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.217904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.217886    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.218037 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.217915    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn5qj\" (UniqueName: \"kubernetes.io/projected/38e72a1b-e54c-4d04-af1a-f35ae2044a56-kube-api-access-mn5qj\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.319214 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.319125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.319214 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.319174    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296hj\" (UniqueName: \"kubernetes.io/projected/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-kube-api-access-296hj\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.319214 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.319201    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.319490 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.319248    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5qj\" (UniqueName: \"kubernetes.io/projected/38e72a1b-e54c-4d04-af1a-f35ae2044a56-kube-api-access-mn5qj\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.319490 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.319297    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.319490 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.319363    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-webhook-cert\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.319490 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:03:21.319368    2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 15 19:03:21.319490 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:03:21.319434    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs podName:38e72a1b-e54c-4d04-af1a-f35ae2044a56 nodeName:}" failed. No retries permitted until 2026-09-15 19:03:21.819415233 +0000 UTC m=+760.675268479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs") pod "maas-controller-5c9bcfd8cf-ndqln" (UID: "38e72a1b-e54c-4d04-af1a-f35ae2044a56") : secret "maas-controller-metrics-tls" not found
Sep 15 19:03:21.319490 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:03:21.319455    2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 15 19:03:21.319823 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:03:21.319539    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs podName:2dfacf49-7e8c-4f07-92fa-44924bc94fb0 nodeName:}" failed. No retries permitted until 2026-09-15 19:03:21.819521046 +0000 UTC m=+760.675374296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs") pod "maas-controller-75dc6cbdf-9mjmn" (UID: "2dfacf49-7e8c-4f07-92fa-44924bc94fb0") : secret "maas-controller-metrics-tls" not found
Sep 15 19:03:21.321678 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.321656    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.321812 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.321794    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-webhook-cert\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.329503 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.329480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5qj\" (UniqueName: \"kubernetes.io/projected/38e72a1b-e54c-4d04-af1a-f35ae2044a56-kube-api-access-mn5qj\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.336430 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.336398    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-296hj\" (UniqueName: \"kubernetes.io/projected/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-kube-api-access-296hj\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.824358 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.824323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:21.824536 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.824396    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.826893 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.826868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-ndqln\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") " pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:21.827087 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:21.827065    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs\") pod \"maas-controller-75dc6cbdf-9mjmn\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") " pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:22.060368 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:22.060332    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:22.078465 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:22.078392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:22.402177 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:22.402154    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-9mjmn"]
Sep 15 19:03:22.404450 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:03:22.404421    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfacf49_7e8c_4f07_92fa_44924bc94fb0.slice/crio-27e5fbafec4e7c054ff045397fe67d1765780d52edd1388fd4d002256b6e79cb WatchSource:0}: Error finding container 27e5fbafec4e7c054ff045397fe67d1765780d52edd1388fd4d002256b6e79cb: Status 404 returned error can't find the container with id 27e5fbafec4e7c054ff045397fe67d1765780d52edd1388fd4d002256b6e79cb
Sep 15 19:03:22.433206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:22.433184    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-ndqln"]
Sep 15 19:03:22.434563 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:03:22.434538    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e72a1b_e54c_4d04_af1a_f35ae2044a56.slice/crio-6ce9f34d6a67dbdc444f08859c91e07fe66e07f3c378a58f3ed333d3dc6b084a WatchSource:0}: Error finding container 6ce9f34d6a67dbdc444f08859c91e07fe66e07f3c378a58f3ed333d3dc6b084a: Status 404 returned error can't find the container with id 6ce9f34d6a67dbdc444f08859c91e07fe66e07f3c378a58f3ed333d3dc6b084a
Sep 15 19:03:22.800977 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:22.800889    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln" event={"ID":"38e72a1b-e54c-4d04-af1a-f35ae2044a56","Type":"ContainerStarted","Data":"6ce9f34d6a67dbdc444f08859c91e07fe66e07f3c378a58f3ed333d3dc6b084a"}
Sep 15 19:03:22.802014 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:22.801991    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn" event={"ID":"2dfacf49-7e8c-4f07-92fa-44924bc94fb0","Type":"ContainerStarted","Data":"27e5fbafec4e7c054ff045397fe67d1765780d52edd1388fd4d002256b6e79cb"}
Sep 15 19:03:25.818466 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:25.818424    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln" event={"ID":"38e72a1b-e54c-4d04-af1a-f35ae2044a56","Type":"ContainerStarted","Data":"59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b"}
Sep 15 19:03:25.818896 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:25.818617    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:25.819969 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:25.819944    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn" event={"ID":"2dfacf49-7e8c-4f07-92fa-44924bc94fb0","Type":"ContainerStarted","Data":"104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d"}
Sep 15 19:03:25.820056 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:25.820000    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:25.835932 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:25.835891    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln" podStartSLOduration=2.493079074 podStartE2EDuration="4.835878232s" totalImagesPullingTime="2.342799158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:25.834358301 +0000 UTC m=+764.690211605" watchObservedRunningTime="2026-09-15 19:03:25.835878232 +0000 UTC m=+764.691731499"
Sep 15 19:03:25.851056 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:25.851011    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn" podStartSLOduration=3.478440161 podStartE2EDuration="5.850997558s" totalImagesPullingTime="2.372557397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:25.849013538 +0000 UTC m=+764.704866807" watchObservedRunningTime="2026-09-15 19:03:25.850997558 +0000 UTC m=+764.706850826"
Sep 15 19:03:36.827071 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:36.827041    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:03:36.827520 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:36.827211    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:36.867432 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:36.867401    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-9mjmn"]
Sep 15 19:03:36.875375 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:36.875334    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn" podUID="2dfacf49-7e8c-4f07-92fa-44924bc94fb0" containerName="manager" containerID="cri-o://104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d" gracePeriod=30
Sep 15 19:03:37.125936 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.125907    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:37.255536 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.255504    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-296hj\" (UniqueName: \"kubernetes.io/projected/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-kube-api-access-296hj\") pod \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") "
Sep 15 19:03:37.255734 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.255617    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-webhook-cert\") pod \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") "
Sep 15 19:03:37.255734 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.255682    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs\") pod \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\" (UID: \"2dfacf49-7e8c-4f07-92fa-44924bc94fb0\") "
Sep 15 19:03:37.257679 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.257630    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2dfacf49-7e8c-4f07-92fa-44924bc94fb0" (UID: "2dfacf49-7e8c-4f07-92fa-44924bc94fb0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:03:37.257780 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.257739    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-kube-api-access-296hj" (OuterVolumeSpecName: "kube-api-access-296hj") pod "2dfacf49-7e8c-4f07-92fa-44924bc94fb0" (UID: "2dfacf49-7e8c-4f07-92fa-44924bc94fb0"). InnerVolumeSpecName "kube-api-access-296hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:03:37.257780 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.257756    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2dfacf49-7e8c-4f07-92fa-44924bc94fb0" (UID: "2dfacf49-7e8c-4f07-92fa-44924bc94fb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:03:37.357017 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.356954    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-296hj\" (UniqueName: \"kubernetes.io/projected/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-kube-api-access-296hj\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:03:37.357017 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.356979    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-webhook-cert\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:03:37.357017 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.356988    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfacf49-7e8c-4f07-92fa-44924bc94fb0-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:03:37.881618 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.881585    2556 generic.go:356] "Generic (PLEG): container finished" podID="2dfacf49-7e8c-4f07-92fa-44924bc94fb0" containerID="104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d" exitCode=0
Sep 15 19:03:37.882099 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.881636    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn"
Sep 15 19:03:37.882099 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.881637    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn" event={"ID":"2dfacf49-7e8c-4f07-92fa-44924bc94fb0","Type":"ContainerDied","Data":"104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d"}
Sep 15 19:03:37.882099 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.881774    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-9mjmn" event={"ID":"2dfacf49-7e8c-4f07-92fa-44924bc94fb0","Type":"ContainerDied","Data":"27e5fbafec4e7c054ff045397fe67d1765780d52edd1388fd4d002256b6e79cb"}
Sep 15 19:03:37.882099 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.881795    2556 scope.go:117] "RemoveContainer" containerID="104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d"
Sep 15 19:03:37.890376 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.890348    2556 scope.go:117] "RemoveContainer" containerID="104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d"
Sep 15 19:03:37.890669 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:03:37.890636    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d\": container with ID starting with 104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d not found: ID does not exist" containerID="104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d"
Sep 15 19:03:37.890719 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.890679    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d"} err="failed to get container status \"104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d\": rpc error: code = NotFound desc = could not find container \"104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d\": container with ID starting with 104e299898f35891b622813e38b824b4b31eef3414a750792dd19873bb6d114d not found: ID does not exist"
Sep 15 19:03:37.896336 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.896315    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-9mjmn"]
Sep 15 19:03:37.900108 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:37.900088    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-9mjmn"]
Sep 15 19:03:39.677863 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:39.677818    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfacf49-7e8c-4f07-92fa-44924bc94fb0" path="/var/lib/kubelet/pods/2dfacf49-7e8c-4f07-92fa-44924bc94fb0/volumes"
Sep 15 19:03:59.334062 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.334024    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"]
Sep 15 19:03:59.334510 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.334380    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfacf49-7e8c-4f07-92fa-44924bc94fb0" containerName="manager"
Sep 15 19:03:59.334510 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.334391    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfacf49-7e8c-4f07-92fa-44924bc94fb0" containerName="manager"
Sep 15 19:03:59.334510 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.334479    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfacf49-7e8c-4f07-92fa-44924bc94fb0" containerName="manager"
Sep 15 19:03:59.340767 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.340746    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.343155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.343134    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v55qf\""
Sep 15 19:03:59.343273 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.343135    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 19:03:59.344131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.344110    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 19:03:59.346739 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.346716    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"]
Sep 15 19:03:59.444914 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.444885    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-maas-api-tls\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.445053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.444928    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-metrics-certs\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.445053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.444964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz5dm\" (UniqueName: \"kubernetes.io/projected/56dff1f4-d575-4103-870e-9de2dda61b67-kube-api-access-mz5dm\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.445053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.445039    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56dff1f4-d575-4103-870e-9de2dda61b67-openshift-service-ca\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.545659 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.545612    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56dff1f4-d575-4103-870e-9de2dda61b67-openshift-service-ca\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.545797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.545719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-maas-api-tls\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.545797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.545753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-metrics-certs\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.545904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.545784    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz5dm\" (UniqueName: \"kubernetes.io/projected/56dff1f4-d575-4103-870e-9de2dda61b67-kube-api-access-mz5dm\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.546405 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.546378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56dff1f4-d575-4103-870e-9de2dda61b67-openshift-service-ca\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.548375 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.548351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-metrics-certs\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.548493 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.548374    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-maas-api-tls\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.558221 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.558197    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz5dm\" (UniqueName: \"kubernetes.io/projected/56dff1f4-d575-4103-870e-9de2dda61b67-kube-api-access-mz5dm\") pod \"maas-api-59f6566bbc-95b72\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.653605 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.653574    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:03:59.810298 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.810264    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"]
Sep 15 19:03:59.813965 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.813943    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:03:59.816318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.816293    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 19:03:59.823909 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.823887    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"]
Sep 15 19:03:59.838314 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.838289    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"]
Sep 15 19:03:59.843528 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.843509    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:03:59.845851 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.845831    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5z5d2\""
Sep 15 19:03:59.861919 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.861893    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"]
Sep 15 19:03:59.928460 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.928407    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"]
Sep 15 19:03:59.931181 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:03:59.931155    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dff1f4_d575_4103_870e_9de2dda61b67.slice/crio-c3269ad1af1e2adb98d72f1681fb2b6ecee97034d4bdace546e59cb99f9cd82c WatchSource:0}: Error finding container c3269ad1af1e2adb98d72f1681fb2b6ecee97034d4bdace546e59cb99f9cd82c: Status 404 returned error can't find the container with id c3269ad1af1e2adb98d72f1681fb2b6ecee97034d4bdace546e59cb99f9cd82c
Sep 15 19:03:59.948612 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.948589    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfjt\" (UniqueName: \"kubernetes.io/projected/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-kube-api-access-rjfjt\") pod \"payload-processing-5b54b7fbb7-bqn5b\" (UID: \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:03:59.948715 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.948663    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bqn5b\" (UID: \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:03:59.948715 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.948693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-f4f46\" (UID: \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:03:59.948808 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.948729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpxv\" (UniqueName: \"kubernetes.io/projected/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-kube-api-access-7rpxv\") pod \"payload-pre-processing-59fbc47bf8-f4f46\" (UID: \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:03:59.985320 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:03:59.985291    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72" event={"ID":"56dff1f4-d575-4103-870e-9de2dda61b67","Type":"ContainerStarted","Data":"c3269ad1af1e2adb98d72f1681fb2b6ecee97034d4bdace546e59cb99f9cd82c"}
Sep 15 19:04:00.049550 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.049514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bqn5b\" (UID: \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:04:00.049769 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.049558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-f4f46\" (UID: \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:04:00.049769 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.049595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpxv\" (UniqueName: \"kubernetes.io/projected/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-kube-api-access-7rpxv\") pod \"payload-pre-processing-59fbc47bf8-f4f46\" (UID: \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:04:00.049769 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.049635    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfjt\" (UniqueName: \"kubernetes.io/projected/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-kube-api-access-rjfjt\") pod \"payload-processing-5b54b7fbb7-bqn5b\" (UID: \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:04:00.050335 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.050310    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bqn5b\" (UID: \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:04:00.050459 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.050437    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-f4f46\" (UID: \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:04:00.057426 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.057403    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpxv\" (UniqueName: \"kubernetes.io/projected/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-kube-api-access-7rpxv\") pod \"payload-pre-processing-59fbc47bf8-f4f46\" (UID: \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:04:00.057508 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.057447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfjt\" (UniqueName: \"kubernetes.io/projected/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-kube-api-access-rjfjt\") pod \"payload-processing-5b54b7fbb7-bqn5b\" (UID: \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:04:00.126422 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.126379    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:04:00.154148 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.154104    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:04:00.399310 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.399286    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"]
Sep 15 19:04:00.402345 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:04:00.402315    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b906d5a_672a_4fbe_ad10_6c7a1e703c49.slice/crio-70cb92540565086475726893316a2703bdb86df09234c486bc7cbbb3719cae64 WatchSource:0}: Error finding container 70cb92540565086475726893316a2703bdb86df09234c486bc7cbbb3719cae64: Status 404 returned error can't find the container with id 70cb92540565086475726893316a2703bdb86df09234c486bc7cbbb3719cae64
Sep 15 19:04:00.436616 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.436420    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"]
Sep 15 19:04:00.438757 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:04:00.438722    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0bc936_379a_46d6_9442_d0c4bac5b8b0.slice/crio-f5761bc3635d88f0abf778f7d6f3d9e423a39a8b5ead7793843fb38393519a33 WatchSource:0}: Error finding container f5761bc3635d88f0abf778f7d6f3d9e423a39a8b5ead7793843fb38393519a33: Status 404 returned error can't find the container with id f5761bc3635d88f0abf778f7d6f3d9e423a39a8b5ead7793843fb38393519a33
Sep 15 19:04:00.994421 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.994200    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46" event={"ID":"3b906d5a-672a-4fbe-ad10-6c7a1e703c49","Type":"ContainerStarted","Data":"70cb92540565086475726893316a2703bdb86df09234c486bc7cbbb3719cae64"}
Sep 15 19:04:00.995951 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:00.995883    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b" event={"ID":"6e0bc936-379a-46d6-9442-d0c4bac5b8b0","Type":"ContainerStarted","Data":"f5761bc3635d88f0abf778f7d6f3d9e423a39a8b5ead7793843fb38393519a33"}
Sep 15 19:04:03.006908 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.006868    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72" event={"ID":"56dff1f4-d575-4103-870e-9de2dda61b67","Type":"ContainerStarted","Data":"32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5"}
Sep 15 19:04:03.007344 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.006935    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:04:03.008482 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.008457    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46" event={"ID":"3b906d5a-672a-4fbe-ad10-6c7a1e703c49","Type":"ContainerStarted","Data":"fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067"}
Sep 15 19:04:03.008594 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.008516    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:04:03.010038 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.010014    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b" event={"ID":"6e0bc936-379a-46d6-9442-d0c4bac5b8b0","Type":"ContainerStarted","Data":"fc9db2d7d1ca3d10082b881a09dec7a6ae9c7a010c28a0c8b0a043dc45e13334"}
Sep 15 19:04:03.010156 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.010085    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:04:03.024389 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.024333    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72" podStartSLOduration=1.8075857229999999 podStartE2EDuration="4.024317584s" totalImagesPullingTime="2.216731861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:03.021932359 +0000 UTC m=+801.877785638" watchObservedRunningTime="2026-09-15 19:04:03.024317584 +0000 UTC m=+801.880170851"
Sep 15 19:04:03.037675 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.037612    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46" podStartSLOduration=2.290546653 podStartE2EDuration="4.037596116s" totalImagesPullingTime="1.747049463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:03.03554697 +0000 UTC m=+801.891400238" watchObservedRunningTime="2026-09-15 19:04:03.037596116 +0000 UTC m=+801.893449383"
Sep 15 19:04:03.051177 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:03.051126    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b" podStartSLOduration=2.342495317 podStartE2EDuration="4.051110805s" totalImagesPullingTime="1.708615488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:03.050097812 +0000 UTC m=+801.905951081" watchObservedRunningTime="2026-09-15 19:04:03.051110805 +0000 UTC m=+801.906964067"
Sep 15 19:04:09.020841 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:09.020759    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:04:14.017227 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:14.017191    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:04:14.017667 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:14.017273    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:04:32.904436 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:32.904404    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"]
Sep 15 19:04:32.908187 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:32.908170    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:32.911502 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:32.911482    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bttfv\""
Sep 15 19:04:32.912768 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:32.912753    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 19:04:32.912978 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:32.912960    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 19:04:32.913029 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:32.913005    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 19:04:32.920256 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:32.920235    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"]
Sep 15 19:04:33.038122 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.038090    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.038317 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.038141    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.038317 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.038251    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26nx9\" (UniqueName: \"kubernetes.io/projected/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-kube-api-access-26nx9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.038317 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.038287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.038457 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.038349    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.038457 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.038387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139180 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139147    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139192    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26nx9\" (UniqueName: \"kubernetes.io/projected/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-kube-api-access-26nx9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139210    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139237    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139338    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139600 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139574    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139667 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139615    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.139707 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.139665    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.141593 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.141568    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.141853 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.141837    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.147585 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.147567    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26nx9\" (UniqueName: \"kubernetes.io/projected/a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b-kube-api-access-26nx9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b\" (UID: \"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.218398 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.218339    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:33.522682 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:33.522657    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"]
Sep 15 19:04:33.524789 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:04:33.524761    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71eebe4_23fe_4d33_a33b_b8bcdafa4e5b.slice/crio-d4f70f583409b9ceb45b3f9089eaa01e658e32be8885efa74fede5f03677dd36 WatchSource:0}: Error finding container d4f70f583409b9ceb45b3f9089eaa01e658e32be8885efa74fede5f03677dd36: Status 404 returned error can't find the container with id d4f70f583409b9ceb45b3f9089eaa01e658e32be8885efa74fede5f03677dd36
Sep 15 19:04:34.170697 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:34.170658    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b" event={"ID":"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b","Type":"ContainerStarted","Data":"d4f70f583409b9ceb45b3f9089eaa01e658e32be8885efa74fede5f03677dd36"}
Sep 15 19:04:35.177699 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:35.177661    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b" event={"ID":"a71eebe4-23fe-4d33-a33b-b8bcdafa4e5b","Type":"ContainerStarted","Data":"3ccab3e41345df4b336ad13a88847ab2d441e14b0d4338a9e63d9f6e85aa3eff"}
Sep 15 19:04:35.198511 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:35.198446    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b" podStartSLOduration=2.161592637 podStartE2EDuration="3.198424464s" totalImagesPullingTime="1.036831827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:35.195881838 +0000 UTC m=+834.051735107" watchObservedRunningTime="2026-09-15 19:04:35.198424464 +0000 UTC m=+834.054277730"
Sep 15 19:04:43.218581 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:43.218548    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:43.219098 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:43.218608    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:43.231198 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:43.231177    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:04:44.231196 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:04:44.231168    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44n7b"
Sep 15 19:05:24.576918 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.576885    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"]
Sep 15 19:05:24.581278 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.581256    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.583777 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.583758    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 19:05:24.588196 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.588173    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"]
Sep 15 19:05:24.708410 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.708383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.708579 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.708422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.708579 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.708456    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.708579 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.708563    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.708792 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.708614    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6lx\" (UniqueName: \"kubernetes.io/projected/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-kube-api-access-kw6lx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.708792 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.708697    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809242 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809210    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809242 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809246    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809409 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809269    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809409 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809301    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809409 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809324    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6lx\" (UniqueName: \"kubernetes.io/projected/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-kube-api-access-kw6lx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809409 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809343    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809769 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809747    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809835 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809789    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.809835 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.809815    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.811435 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.811416    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.811905 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.811889    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.816917 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.816898    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6lx\" (UniqueName: \"kubernetes.io/projected/32f4eac3-55ce-4507-9cfc-e20608ddc8a7-kube-api-access-kw6lx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j\" (UID: \"32f4eac3-55ce-4507-9cfc-e20608ddc8a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:24.892686 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:24.892658    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:25.161340 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:25.161316    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"]
Sep 15 19:05:25.163571 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:05:25.163540    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f4eac3_55ce_4507_9cfc_e20608ddc8a7.slice/crio-3b8df0731ee857198c94be9c381d4d9cc2b185e9d5130e998416d7b5c127772d WatchSource:0}: Error finding container 3b8df0731ee857198c94be9c381d4d9cc2b185e9d5130e998416d7b5c127772d: Status 404 returned error can't find the container with id 3b8df0731ee857198c94be9c381d4d9cc2b185e9d5130e998416d7b5c127772d
Sep 15 19:05:25.431502 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:25.431474    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j" event={"ID":"32f4eac3-55ce-4507-9cfc-e20608ddc8a7","Type":"ContainerStarted","Data":"a0d0b8fdad6d734586a3cacaa2ed85c58dc4429d16fcc327134a1ad05250cbfc"}
Sep 15 19:05:25.431619 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:25.431531    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j" event={"ID":"32f4eac3-55ce-4507-9cfc-e20608ddc8a7","Type":"ContainerStarted","Data":"3b8df0731ee857198c94be9c381d4d9cc2b185e9d5130e998416d7b5c127772d"}
Sep 15 19:05:25.446885 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:25.446840    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j" podStartSLOduration=1.2454910479999999 podStartE2EDuration="1.4468245s" totalImagesPullingTime="201.333452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:25.445681535 +0000 UTC m=+884.301534803" watchObservedRunningTime="2026-09-15 19:05:25.4468245 +0000 UTC m=+884.302677768"
Sep 15 19:05:34.893139 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:34.893046    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:34.893139 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:34.893135    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:34.906143 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:34.906117    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:35.491153 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:35.491120    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm769j"
Sep 15 19:05:41.708436 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:41.708343    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:05:41.709547 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:41.709039    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:05:41.713363 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:41.713340    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:05:41.714033 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:05:41.714014    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:06:38.249581 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.249543    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d854f4d76-l8kht"]
Sep 15 19:06:38.253188 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.253167    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d854f4d76-l8kht"
Sep 15 19:06:38.255410 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.255386    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 19:06:38.256445 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.256422    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 19:06:38.256445 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.256436    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 19:06:38.256592 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.256422    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mzzjb\""
Sep 15 19:06:38.260038 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.260011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24lgd\" (UniqueName: \"kubernetes.io/projected/6e1f175f-98f3-4c0b-8872-ea716c90656c-kube-api-access-24lgd\") pod \"authorino-6d854f4d76-l8kht\" (UID: \"6e1f175f-98f3-4c0b-8872-ea716c90656c\") " pod="kuadrant-system/authorino-6d854f4d76-l8kht"
Sep 15 19:06:38.260145 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.260067    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e1f175f-98f3-4c0b-8872-ea716c90656c-tls-cert\") pod \"authorino-6d854f4d76-l8kht\" (UID: \"6e1f175f-98f3-4c0b-8872-ea716c90656c\") " pod="kuadrant-system/authorino-6d854f4d76-l8kht"
Sep 15 19:06:38.261005 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.260813    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d854f4d76-l8kht"]
Sep 15 19:06:38.360753 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.360725    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24lgd\" (UniqueName: \"kubernetes.io/projected/6e1f175f-98f3-4c0b-8872-ea716c90656c-kube-api-access-24lgd\") pod \"authorino-6d854f4d76-l8kht\" (UID: \"6e1f175f-98f3-4c0b-8872-ea716c90656c\") " pod="kuadrant-system/authorino-6d854f4d76-l8kht"
Sep 15 19:06:38.360910 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.360768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e1f175f-98f3-4c0b-8872-ea716c90656c-tls-cert\") pod \"authorino-6d854f4d76-l8kht\" (UID: \"6e1f175f-98f3-4c0b-8872-ea716c90656c\") " pod="kuadrant-system/authorino-6d854f4d76-l8kht"
Sep 15 19:06:38.363280 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.363249    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e1f175f-98f3-4c0b-8872-ea716c90656c-tls-cert\") pod \"authorino-6d854f4d76-l8kht\" (UID: \"6e1f175f-98f3-4c0b-8872-ea716c90656c\") " pod="kuadrant-system/authorino-6d854f4d76-l8kht"
Sep 15 19:06:38.368017 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.368000    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24lgd\" (UniqueName: \"kubernetes.io/projected/6e1f175f-98f3-4c0b-8872-ea716c90656c-kube-api-access-24lgd\") pod \"authorino-6d854f4d76-l8kht\" (UID: \"6e1f175f-98f3-4c0b-8872-ea716c90656c\") " pod="kuadrant-system/authorino-6d854f4d76-l8kht"
Sep 15 19:06:38.563768 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.563676    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d854f4d76-l8kht"
Sep 15 19:06:38.833350 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:38.833320    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d854f4d76-l8kht"]
Sep 15 19:06:38.834889 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:06:38.834859    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e1f175f_98f3_4c0b_8872_ea716c90656c.slice/crio-b0c32ade69a82d5d7330196e36e4fab7edf12445d34c8fc491ad0e5ed63eb7d8 WatchSource:0}: Error finding container b0c32ade69a82d5d7330196e36e4fab7edf12445d34c8fc491ad0e5ed63eb7d8: Status 404 returned error can't find the container with id b0c32ade69a82d5d7330196e36e4fab7edf12445d34c8fc491ad0e5ed63eb7d8
Sep 15 19:06:39.786705 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:39.786676    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d854f4d76-l8kht" event={"ID":"6e1f175f-98f3-4c0b-8872-ea716c90656c","Type":"ContainerStarted","Data":"b0c32ade69a82d5d7330196e36e4fab7edf12445d34c8fc491ad0e5ed63eb7d8"}
Sep 15 19:06:40.794256 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:40.794223    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d854f4d76-l8kht" event={"ID":"6e1f175f-98f3-4c0b-8872-ea716c90656c","Type":"ContainerStarted","Data":"c4bc5bd61c3f024ddb0612031c68d561218b0d5c0fd65a592f6105e7febd0b01"}
Sep 15 19:06:40.809345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:40.809275    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d854f4d76-l8kht" podStartSLOduration=1.9379252519999999 podStartE2EDuration="2.809251535s" totalImagesPullingTime="871.326283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:06:40.807637264 +0000 UTC m=+959.663490546" watchObservedRunningTime="2026-09-15 19:06:40.809251535 +0000 UTC m=+959.665104804"
Sep 15 19:06:41.341265 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.341220    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-ndqln"]
Sep 15 19:06:41.341465 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.341424    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln" podUID="38e72a1b-e54c-4d04-af1a-f35ae2044a56" containerName="manager" containerID="cri-o://59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b" gracePeriod=30
Sep 15 19:06:41.564579 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.564414    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"]
Sep 15 19:06:41.564787 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.564685    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b" podUID="6e0bc936-379a-46d6-9442-d0c4bac5b8b0" containerName="payload-processing" containerID="cri-o://fc9db2d7d1ca3d10082b881a09dec7a6ae9c7a010c28a0c8b0a043dc45e13334" gracePeriod=30
Sep 15 19:06:41.701238 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.700171    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"]
Sep 15 19:06:41.701238 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.700418    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46" podUID="3b906d5a-672a-4fbe-ad10-6c7a1e703c49" containerName="payload-pre-processing" containerID="cri-o://fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067" gracePeriod=30
Sep 15 19:06:41.718454 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.718420    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:06:41.798498 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.797927    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-webhook-cert\") pod \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") "
Sep 15 19:06:41.798498 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.797985    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs\") pod \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") "
Sep 15 19:06:41.798498 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.798088    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn5qj\" (UniqueName: \"kubernetes.io/projected/38e72a1b-e54c-4d04-af1a-f35ae2044a56-kube-api-access-mn5qj\") pod \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\" (UID: \"38e72a1b-e54c-4d04-af1a-f35ae2044a56\") "
Sep 15 19:06:41.802869 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.802838    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "38e72a1b-e54c-4d04-af1a-f35ae2044a56" (UID: "38e72a1b-e54c-4d04-af1a-f35ae2044a56"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:06:41.803040 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.802863    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e72a1b-e54c-4d04-af1a-f35ae2044a56-kube-api-access-mn5qj" (OuterVolumeSpecName: "kube-api-access-mn5qj") pod "38e72a1b-e54c-4d04-af1a-f35ae2044a56" (UID: "38e72a1b-e54c-4d04-af1a-f35ae2044a56"). InnerVolumeSpecName "kube-api-access-mn5qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:06:41.803294 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.803267    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "38e72a1b-e54c-4d04-af1a-f35ae2044a56" (UID: "38e72a1b-e54c-4d04-af1a-f35ae2044a56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:06:41.807949 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.807919    2556 generic.go:356] "Generic (PLEG): container finished" podID="6e0bc936-379a-46d6-9442-d0c4bac5b8b0" containerID="fc9db2d7d1ca3d10082b881a09dec7a6ae9c7a010c28a0c8b0a043dc45e13334" exitCode=0
Sep 15 19:06:41.808127 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.808097    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b" event={"ID":"6e0bc936-379a-46d6-9442-d0c4bac5b8b0","Type":"ContainerDied","Data":"fc9db2d7d1ca3d10082b881a09dec7a6ae9c7a010c28a0c8b0a043dc45e13334"}
Sep 15 19:06:41.810233 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.810213    2556 generic.go:356] "Generic (PLEG): container finished" podID="38e72a1b-e54c-4d04-af1a-f35ae2044a56" containerID="59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b" exitCode=0
Sep 15 19:06:41.810448 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.810290    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln" event={"ID":"38e72a1b-e54c-4d04-af1a-f35ae2044a56","Type":"ContainerDied","Data":"59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b"}
Sep 15 19:06:41.810632 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.810324    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln"
Sep 15 19:06:41.810822 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.810668    2556 scope.go:117] "RemoveContainer" containerID="59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b"
Sep 15 19:06:41.811594 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.810609    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-ndqln" event={"ID":"38e72a1b-e54c-4d04-af1a-f35ae2044a56","Type":"ContainerDied","Data":"6ce9f34d6a67dbdc444f08859c91e07fe66e07f3c378a58f3ed333d3dc6b084a"}
Sep 15 19:06:41.830027 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.830005    2556 scope.go:117] "RemoveContainer" containerID="59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b"
Sep 15 19:06:41.830425 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:06:41.830392    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b\": container with ID starting with 59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b not found: ID does not exist" containerID="59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b"
Sep 15 19:06:41.830501 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.830447    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b"} err="failed to get container status \"59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b\": rpc error: code = NotFound desc = could not find container \"59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b\": container with ID starting with 59c70d20b6a223ee2fba6c93033ac0c91a5922310c329620af193c4925f35c0b not found: ID does not exist"
Sep 15 19:06:41.839745 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.839725    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-ndqln"]
Sep 15 19:06:41.844394 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.844370    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-ndqln"]
Sep 15 19:06:41.847149 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.847128    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:06:41.899162 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.899128    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjfjt\" (UniqueName: \"kubernetes.io/projected/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-kube-api-access-rjfjt\") pod \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\" (UID: \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\") "
Sep 15 19:06:41.899374 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.899169    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-plugins-config-volume\") pod \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\" (UID: \"6e0bc936-379a-46d6-9442-d0c4bac5b8b0\") "
Sep 15 19:06:41.899565 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.899544    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-webhook-cert\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:06:41.899719 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.899692    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e0bc936-379a-46d6-9442-d0c4bac5b8b0" (UID: "6e0bc936-379a-46d6-9442-d0c4bac5b8b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:06:41.899827 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.899813    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38e72a1b-e54c-4d04-af1a-f35ae2044a56-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:06:41.899919 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.899907    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn5qj\" (UniqueName: \"kubernetes.io/projected/38e72a1b-e54c-4d04-af1a-f35ae2044a56-kube-api-access-mn5qj\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:06:41.904492 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.904465    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-kube-api-access-rjfjt" (OuterVolumeSpecName: "kube-api-access-rjfjt") pod "6e0bc936-379a-46d6-9442-d0c4bac5b8b0" (UID: "6e0bc936-379a-46d6-9442-d0c4bac5b8b0"). InnerVolumeSpecName "kube-api-access-rjfjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:06:41.967552 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:41.967527    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:06:42.001199 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.001171    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rpxv\" (UniqueName: \"kubernetes.io/projected/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-kube-api-access-7rpxv\") pod \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\" (UID: \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\") "
Sep 15 19:06:42.001374 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.001256    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-plugins-config-volume\") pod \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\" (UID: \"3b906d5a-672a-4fbe-ad10-6c7a1e703c49\") "
Sep 15 19:06:42.001525 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.001503    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfjt\" (UniqueName: \"kubernetes.io/projected/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-kube-api-access-rjfjt\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:06:42.001525 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.001522    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0bc936-379a-46d6-9442-d0c4bac5b8b0-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:06:42.001759 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.001728    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b906d5a-672a-4fbe-ad10-6c7a1e703c49" (UID: "3b906d5a-672a-4fbe-ad10-6c7a1e703c49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:06:42.003386 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.003360    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-kube-api-access-7rpxv" (OuterVolumeSpecName: "kube-api-access-7rpxv") pod "3b906d5a-672a-4fbe-ad10-6c7a1e703c49" (UID: "3b906d5a-672a-4fbe-ad10-6c7a1e703c49"). InnerVolumeSpecName "kube-api-access-7rpxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:06:42.102335 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.102291    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:06:42.102335 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.102330    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rpxv\" (UniqueName: \"kubernetes.io/projected/3b906d5a-672a-4fbe-ad10-6c7a1e703c49-kube-api-access-7rpxv\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:06:42.355527 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355446    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-v74sg"]
Sep 15 19:06:42.355857 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355844    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b906d5a-672a-4fbe-ad10-6c7a1e703c49" containerName="payload-pre-processing"
Sep 15 19:06:42.355904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355859    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b906d5a-672a-4fbe-ad10-6c7a1e703c49" containerName="payload-pre-processing"
Sep 15 19:06:42.355904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355873    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e0bc936-379a-46d6-9442-d0c4bac5b8b0" containerName="payload-processing"
Sep 15 19:06:42.355904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355879    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e0bc936-379a-46d6-9442-d0c4bac5b8b0" containerName="payload-processing"
Sep 15 19:06:42.355904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355890    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e72a1b-e54c-4d04-af1a-f35ae2044a56" containerName="manager"
Sep 15 19:06:42.355904 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355896    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e72a1b-e54c-4d04-af1a-f35ae2044a56" containerName="manager"
Sep 15 19:06:42.356053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355959    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b906d5a-672a-4fbe-ad10-6c7a1e703c49" containerName="payload-pre-processing"
Sep 15 19:06:42.356053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355968    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e0bc936-379a-46d6-9442-d0c4bac5b8b0" containerName="payload-processing"
Sep 15 19:06:42.356053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.355976    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e72a1b-e54c-4d04-af1a-f35ae2044a56" containerName="manager"
Sep 15 19:06:42.359015 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.358999    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.361692 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.361669    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vc5j9\""
Sep 15 19:06:42.361692 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.361687    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 19:06:42.362472 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.362455    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 19:06:42.362558 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.362471    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 19:06:42.368673 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.368623    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-v74sg"]
Sep 15 19:06:42.405482 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.405453    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.405584 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.405494    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.405662 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.405607    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bqdk\" (UniqueName: \"kubernetes.io/projected/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-kube-api-access-9bqdk\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.506183 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.506149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bqdk\" (UniqueName: \"kubernetes.io/projected/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-kube-api-access-9bqdk\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.506311 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.506213    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.506311 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.506257    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.508732 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.508705    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.508852 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.508735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.514485 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.514461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bqdk\" (UniqueName: \"kubernetes.io/projected/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-kube-api-access-9bqdk\") pod \"maas-controller-5c9bcfd8cf-v74sg\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") " pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.686072 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.686043    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:42.817909 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.817876    2556 generic.go:356] "Generic (PLEG): container finished" podID="3b906d5a-672a-4fbe-ad10-6c7a1e703c49" containerID="fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067" exitCode=0
Sep 15 19:06:42.818345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.817924    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"
Sep 15 19:06:42.818345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.817964    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46" event={"ID":"3b906d5a-672a-4fbe-ad10-6c7a1e703c49","Type":"ContainerDied","Data":"fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067"}
Sep 15 19:06:42.818345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.818028    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46" event={"ID":"3b906d5a-672a-4fbe-ad10-6c7a1e703c49","Type":"ContainerDied","Data":"70cb92540565086475726893316a2703bdb86df09234c486bc7cbbb3719cae64"}
Sep 15 19:06:42.818345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.818066    2556 scope.go:117] "RemoveContainer" containerID="fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067"
Sep 15 19:06:42.819525 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.819501    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b" event={"ID":"6e0bc936-379a-46d6-9442-d0c4bac5b8b0","Type":"ContainerDied","Data":"f5761bc3635d88f0abf778f7d6f3d9e423a39a8b5ead7793843fb38393519a33"}
Sep 15 19:06:42.819525 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.819519    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"
Sep 15 19:06:42.826450 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.826432    2556 scope.go:117] "RemoveContainer" containerID="fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067"
Sep 15 19:06:42.826734 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:06:42.826705    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067\": container with ID starting with fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067 not found: ID does not exist" containerID="fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067"
Sep 15 19:06:42.826822 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.826739    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067"} err="failed to get container status \"fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067\": rpc error: code = NotFound desc = could not find container \"fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067\": container with ID starting with fab01d8127170bbb87ee2c7ca8c43cbfff381625f441cae1851c175ceea17067 not found: ID does not exist"
Sep 15 19:06:42.826822 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.826758    2556 scope.go:117] "RemoveContainer" containerID="fc9db2d7d1ca3d10082b881a09dec7a6ae9c7a010c28a0c8b0a043dc45e13334"
Sep 15 19:06:42.845823 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.845790    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"]
Sep 15 19:06:42.849060 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.849038    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f4f46"]
Sep 15 19:06:42.859727 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.859694    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"]
Sep 15 19:06:42.861285 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.861267    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bqn5b"]
Sep 15 19:06:42.954887 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:42.954860    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-v74sg"]
Sep 15 19:06:42.956621 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:06:42.956599    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fc7cd2_1ac5_4f23_8e57_fabbc3d6fa63.slice/crio-3d505d1ef43fdbe701f2f8be38a36b4dce5d722dd9033d680ce4a1cbd3a407fb WatchSource:0}: Error finding container 3d505d1ef43fdbe701f2f8be38a36b4dce5d722dd9033d680ce4a1cbd3a407fb: Status 404 returned error can't find the container with id 3d505d1ef43fdbe701f2f8be38a36b4dce5d722dd9033d680ce4a1cbd3a407fb
Sep 15 19:06:43.678370 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:43.678334    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38e72a1b-e54c-4d04-af1a-f35ae2044a56" path="/var/lib/kubelet/pods/38e72a1b-e54c-4d04-af1a-f35ae2044a56/volumes"
Sep 15 19:06:43.678738 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:43.678719    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b906d5a-672a-4fbe-ad10-6c7a1e703c49" path="/var/lib/kubelet/pods/3b906d5a-672a-4fbe-ad10-6c7a1e703c49/volumes"
Sep 15 19:06:43.679053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:43.679042    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e0bc936-379a-46d6-9442-d0c4bac5b8b0" path="/var/lib/kubelet/pods/6e0bc936-379a-46d6-9442-d0c4bac5b8b0/volumes"
Sep 15 19:06:43.826570 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:43.826543    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg" event={"ID":"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63","Type":"ContainerStarted","Data":"3d505d1ef43fdbe701f2f8be38a36b4dce5d722dd9033d680ce4a1cbd3a407fb"}
Sep 15 19:06:45.837695 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:45.837662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg" event={"ID":"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63","Type":"ContainerStarted","Data":"fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f"}
Sep 15 19:06:45.838045 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:45.837768    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:45.855814 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:45.855767    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg" podStartSLOduration=1.10702295 podStartE2EDuration="3.855752781s" totalImagesPullingTime="2.748729831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:06:45.853997609 +0000 UTC m=+964.709850881" watchObservedRunningTime="2026-09-15 19:06:45.855752781 +0000 UTC m=+964.711606051"
Sep 15 19:06:56.845476 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:56.845440    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:06:57.095339 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.095302    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-zhwmm"]
Sep 15 19:06:57.098973 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.098929    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:06:57.101374 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.101353    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-h8m6r\""
Sep 15 19:06:57.101471 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.101353    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 15 19:06:57.106198 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.105974    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-zhwmm"]
Sep 15 19:06:57.232674 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.232623    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n822v\" (UniqueName: \"kubernetes.io/projected/251fc872-19bd-4d0f-a53d-81c2a7fbef12-kube-api-access-n822v\") pod \"ai-gateway-controller-6847588947-zhwmm\" (UID: \"251fc872-19bd-4d0f-a53d-81c2a7fbef12\") " pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:06:57.333542 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.333505    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n822v\" (UniqueName: \"kubernetes.io/projected/251fc872-19bd-4d0f-a53d-81c2a7fbef12-kube-api-access-n822v\") pod \"ai-gateway-controller-6847588947-zhwmm\" (UID: \"251fc872-19bd-4d0f-a53d-81c2a7fbef12\") " pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:06:57.340911 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.340891    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n822v\" (UniqueName: \"kubernetes.io/projected/251fc872-19bd-4d0f-a53d-81c2a7fbef12-kube-api-access-n822v\") pod \"ai-gateway-controller-6847588947-zhwmm\" (UID: \"251fc872-19bd-4d0f-a53d-81c2a7fbef12\") " pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:06:57.409369 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.409336    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:06:57.679866 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.679835    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-zhwmm"]
Sep 15 19:06:57.680108 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:06:57.680086    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251fc872_19bd_4d0f_a53d_81c2a7fbef12.slice/crio-c04927cb6621772b704a727f9ba0ddc94db34911bb18e27e4bf126f14b295af1 WatchSource:0}: Error finding container c04927cb6621772b704a727f9ba0ddc94db34911bb18e27e4bf126f14b295af1: Status 404 returned error can't find the container with id c04927cb6621772b704a727f9ba0ddc94db34911bb18e27e4bf126f14b295af1
Sep 15 19:06:57.904855 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:06:57.904815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm" event={"ID":"251fc872-19bd-4d0f-a53d-81c2a7fbef12","Type":"ContainerStarted","Data":"c04927cb6621772b704a727f9ba0ddc94db34911bb18e27e4bf126f14b295af1"}
Sep 15 19:07:00.880216 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.880175    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-74969cc5f8-hnjvz"]
Sep 15 19:07:00.884578 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.884552    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:00.887102 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.887073    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 19:07:00.887102 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.887073    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5z5d2\""
Sep 15 19:07:00.891710 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.891688    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-74969cc5f8-hnjvz"]
Sep 15 19:07:00.909578 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.909552    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"]
Sep 15 19:07:00.913269 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.913250    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:00.922711 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.922610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"]
Sep 15 19:07:00.928613 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.928580    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm" event={"ID":"251fc872-19bd-4d0f-a53d-81c2a7fbef12","Type":"ContainerStarted","Data":"82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba"}
Sep 15 19:07:00.928743 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.928689    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:07:00.948625 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.948579    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm" podStartSLOduration=1.152893959 podStartE2EDuration="3.948564239s" totalImagesPullingTime="2.79567028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:00.947306085 +0000 UTC m=+979.803159354" watchObservedRunningTime="2026-09-15 19:07:00.948564239 +0000 UTC m=+979.804417508"
Sep 15 19:07:00.968323 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.968290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq98l\" (UniqueName: \"kubernetes.io/projected/3b536038-6bba-4a1a-a625-415d0e8712d6-kube-api-access-jq98l\") pod \"payload-processing-74969cc5f8-hnjvz\" (UID: \"3b536038-6bba-4a1a-a625-415d0e8712d6\") " pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:00.968419 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.968340    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa75d43-9279-4356-b3e5-0efaaa9aac59-plugins-config-volume\") pod \"payload-pre-processing-558d9b8847-jqkvg\" (UID: \"3aa75d43-9279-4356-b3e5-0efaaa9aac59\") " pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:00.968419 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.968360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b536038-6bba-4a1a-a625-415d0e8712d6-plugins-config-volume\") pod \"payload-processing-74969cc5f8-hnjvz\" (UID: \"3b536038-6bba-4a1a-a625-415d0e8712d6\") " pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:00.968490 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:00.968424    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwcz7\" (UniqueName: \"kubernetes.io/projected/3aa75d43-9279-4356-b3e5-0efaaa9aac59-kube-api-access-lwcz7\") pod \"payload-pre-processing-558d9b8847-jqkvg\" (UID: \"3aa75d43-9279-4356-b3e5-0efaaa9aac59\") " pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:01.069922 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.069885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa75d43-9279-4356-b3e5-0efaaa9aac59-plugins-config-volume\") pod \"payload-pre-processing-558d9b8847-jqkvg\" (UID: \"3aa75d43-9279-4356-b3e5-0efaaa9aac59\") " pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:01.070087 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.069925    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b536038-6bba-4a1a-a625-415d0e8712d6-plugins-config-volume\") pod \"payload-processing-74969cc5f8-hnjvz\" (UID: \"3b536038-6bba-4a1a-a625-415d0e8712d6\") " pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:01.070087 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.069975    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwcz7\" (UniqueName: \"kubernetes.io/projected/3aa75d43-9279-4356-b3e5-0efaaa9aac59-kube-api-access-lwcz7\") pod \"payload-pre-processing-558d9b8847-jqkvg\" (UID: \"3aa75d43-9279-4356-b3e5-0efaaa9aac59\") " pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:01.070087 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.070034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq98l\" (UniqueName: \"kubernetes.io/projected/3b536038-6bba-4a1a-a625-415d0e8712d6-kube-api-access-jq98l\") pod \"payload-processing-74969cc5f8-hnjvz\" (UID: \"3b536038-6bba-4a1a-a625-415d0e8712d6\") " pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:01.071128 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.071100    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa75d43-9279-4356-b3e5-0efaaa9aac59-plugins-config-volume\") pod \"payload-pre-processing-558d9b8847-jqkvg\" (UID: \"3aa75d43-9279-4356-b3e5-0efaaa9aac59\") " pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:01.071128 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.071120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b536038-6bba-4a1a-a625-415d0e8712d6-plugins-config-volume\") pod \"payload-processing-74969cc5f8-hnjvz\" (UID: \"3b536038-6bba-4a1a-a625-415d0e8712d6\") " pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:01.078082 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.078050    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwcz7\" (UniqueName: \"kubernetes.io/projected/3aa75d43-9279-4356-b3e5-0efaaa9aac59-kube-api-access-lwcz7\") pod \"payload-pre-processing-558d9b8847-jqkvg\" (UID: \"3aa75d43-9279-4356-b3e5-0efaaa9aac59\") " pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:01.078082 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.078077    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq98l\" (UniqueName: \"kubernetes.io/projected/3b536038-6bba-4a1a-a625-415d0e8712d6-kube-api-access-jq98l\") pod \"payload-processing-74969cc5f8-hnjvz\" (UID: \"3b536038-6bba-4a1a-a625-415d0e8712d6\") " pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:01.196746 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.196664    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:01.226093 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.226062    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:01.472275 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.472248    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-74969cc5f8-hnjvz"]
Sep 15 19:07:01.474363 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:01.474329    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b536038_6bba_4a1a_a625_415d0e8712d6.slice/crio-366b14dbe91e6356d6f3b0b3050fc04d071cc8f40c32c28159aa1337b0d0ef84 WatchSource:0}: Error finding container 366b14dbe91e6356d6f3b0b3050fc04d071cc8f40c32c28159aa1337b0d0ef84: Status 404 returned error can't find the container with id 366b14dbe91e6356d6f3b0b3050fc04d071cc8f40c32c28159aa1337b0d0ef84
Sep 15 19:07:01.510704 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.510676    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"]
Sep 15 19:07:01.512099 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:01.512063    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa75d43_9279_4356_b3e5_0efaaa9aac59.slice/crio-ef43b1416e3f6d37623f8c1e7ec6ebfbe8bacad42a07fc8227ac58bcae25972f WatchSource:0}: Error finding container ef43b1416e3f6d37623f8c1e7ec6ebfbe8bacad42a07fc8227ac58bcae25972f: Status 404 returned error can't find the container with id ef43b1416e3f6d37623f8c1e7ec6ebfbe8bacad42a07fc8227ac58bcae25972f
Sep 15 19:07:01.934682 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.934625    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" event={"ID":"3aa75d43-9279-4356-b3e5-0efaaa9aac59","Type":"ContainerStarted","Data":"ef43b1416e3f6d37623f8c1e7ec6ebfbe8bacad42a07fc8227ac58bcae25972f"}
Sep 15 19:07:01.935876 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:01.935851    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" event={"ID":"3b536038-6bba-4a1a-a625-415d0e8712d6","Type":"ContainerStarted","Data":"366b14dbe91e6356d6f3b0b3050fc04d071cc8f40c32c28159aa1337b0d0ef84"}
Sep 15 19:07:03.955891 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:03.955800    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" event={"ID":"3b536038-6bba-4a1a-a625-415d0e8712d6","Type":"ContainerStarted","Data":"019d033adbcbcc190a3b28bda5510b269794ce261881deb6f1b80be5aa25b733"}
Sep 15 19:07:03.956238 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:03.955899    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:03.973139 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:03.973029    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" podStartSLOduration=1.736656408 podStartE2EDuration="3.973012468s" totalImagesPullingTime="2.23635606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:03.972680604 +0000 UTC m=+982.828533874" watchObservedRunningTime="2026-09-15 19:07:03.973012468 +0000 UTC m=+982.828865742"
Sep 15 19:07:04.964270 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:04.964232    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" event={"ID":"3aa75d43-9279-4356-b3e5-0efaaa9aac59","Type":"ContainerStarted","Data":"d4c738b403625b9e4220e39ffcf55a1be17afd04373ccae26b624f3e256afd07"}
Sep 15 19:07:04.964720 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:04.964445    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:04.985410 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:04.985351    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" podStartSLOduration=2.514100915 podStartE2EDuration="4.985332875s" totalImagesPullingTime="2.47123196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:04.983829734 +0000 UTC m=+983.839683012" watchObservedRunningTime="2026-09-15 19:07:04.985332875 +0000 UTC m=+983.841186143"
Sep 15 19:07:05.969926 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:05.969894    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:07:11.938553 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:11.938519    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:07:12.118897 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:12.118863    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"]
Sep 15 19:07:12.122494 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:12.122474    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"
Sep 15 19:07:12.129443 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:12.129417    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"]
Sep 15 19:07:12.278365 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:12.278277    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bdqw\" (UniqueName: \"kubernetes.io/projected/0e0a67bb-859e-4cdf-aeee-5f1c35893446-kube-api-access-9bdqw\") pod \"ai-gateway-controller-747f7b9d6c-r9j59\" (UID: \"0e0a67bb-859e-4cdf-aeee-5f1c35893446\") " pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"
Sep 15 19:07:12.379197 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:12.379151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdqw\" (UniqueName: \"kubernetes.io/projected/0e0a67bb-859e-4cdf-aeee-5f1c35893446-kube-api-access-9bdqw\") pod \"ai-gateway-controller-747f7b9d6c-r9j59\" (UID: \"0e0a67bb-859e-4cdf-aeee-5f1c35893446\") " pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"
Sep 15 19:07:12.387900 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:12.387867    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdqw\" (UniqueName: \"kubernetes.io/projected/0e0a67bb-859e-4cdf-aeee-5f1c35893446-kube-api-access-9bdqw\") pod \"ai-gateway-controller-747f7b9d6c-r9j59\" (UID: \"0e0a67bb-859e-4cdf-aeee-5f1c35893446\") " pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"
Sep 15 19:07:12.434754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:12.434718    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"
Sep 15 19:07:12.703400 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:12.703365    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"]
Sep 15 19:07:12.704857 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:12.704836    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0a67bb_859e_4cdf_aeee_5f1c35893446.slice/crio-2479d90cd1869618df8b192d5f075b8d6699d78f2f820517fb5dd250605dbd7f WatchSource:0}: Error finding container 2479d90cd1869618df8b192d5f075b8d6699d78f2f820517fb5dd250605dbd7f: Status 404 returned error can't find the container with id 2479d90cd1869618df8b192d5f075b8d6699d78f2f820517fb5dd250605dbd7f
Sep 15 19:07:13.005326 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:13.005237    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59" event={"ID":"0e0a67bb-859e-4cdf-aeee-5f1c35893446","Type":"ContainerStarted","Data":"2479d90cd1869618df8b192d5f075b8d6699d78f2f820517fb5dd250605dbd7f"}
Sep 15 19:07:14.012090 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:14.012059    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59" event={"ID":"0e0a67bb-859e-4cdf-aeee-5f1c35893446","Type":"ContainerStarted","Data":"9f0ffec9529cb2e570464b77c8b654c9c3cbf653a5a43902bf5abab872cb2c3d"}
Sep 15 19:07:14.012442 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:14.012116    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"
Sep 15 19:07:14.028178 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:14.028125    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59" podStartSLOduration=1.592522891 podStartE2EDuration="2.028107448s" totalImagesPullingTime="435.584557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:14.027499746 +0000 UTC m=+992.883353014" watchObservedRunningTime="2026-09-15 19:07:14.028107448 +0000 UTC m=+992.883960717"
Sep 15 19:07:14.966978 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:14.966946    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz"
Sep 15 19:07:14.967316 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:14.967300    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:07:15.972359 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:15.972328    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg"
Sep 15 19:07:15.972749 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:15.972736    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:07:25.020694 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:25.020664    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-747f7b9d6c-r9j59"
Sep 15 19:07:25.057845 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:25.057811    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-zhwmm"]
Sep 15 19:07:25.058085 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:25.058031    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm" podUID="251fc872-19bd-4d0f-a53d-81c2a7fbef12" containerName="manager" containerID="cri-o://82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba" gracePeriod=30
Sep 15 19:07:25.306836 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:25.306813    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:07:25.400734 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:25.400706    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n822v\" (UniqueName: \"kubernetes.io/projected/251fc872-19bd-4d0f-a53d-81c2a7fbef12-kube-api-access-n822v\") pod \"251fc872-19bd-4d0f-a53d-81c2a7fbef12\" (UID: \"251fc872-19bd-4d0f-a53d-81c2a7fbef12\") "
Sep 15 19:07:25.402754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:25.402710    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fc872-19bd-4d0f-a53d-81c2a7fbef12-kube-api-access-n822v" (OuterVolumeSpecName: "kube-api-access-n822v") pod "251fc872-19bd-4d0f-a53d-81c2a7fbef12" (UID: "251fc872-19bd-4d0f-a53d-81c2a7fbef12"). InnerVolumeSpecName "kube-api-access-n822v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:07:25.502332 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:25.502309    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n822v\" (UniqueName: \"kubernetes.io/projected/251fc872-19bd-4d0f-a53d-81c2a7fbef12-kube-api-access-n822v\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:07:26.077142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.077060    2556 generic.go:356] "Generic (PLEG): container finished" podID="251fc872-19bd-4d0f-a53d-81c2a7fbef12" containerID="82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba" exitCode=0
Sep 15 19:07:26.077142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.077117    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm"
Sep 15 19:07:26.077568 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.077150    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm" event={"ID":"251fc872-19bd-4d0f-a53d-81c2a7fbef12","Type":"ContainerDied","Data":"82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba"}
Sep 15 19:07:26.077568 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.077209    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6847588947-zhwmm" event={"ID":"251fc872-19bd-4d0f-a53d-81c2a7fbef12","Type":"ContainerDied","Data":"c04927cb6621772b704a727f9ba0ddc94db34911bb18e27e4bf126f14b295af1"}
Sep 15 19:07:26.077568 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.077238    2556 scope.go:117] "RemoveContainer" containerID="82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba"
Sep 15 19:07:26.086214 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.086199    2556 scope.go:117] "RemoveContainer" containerID="82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba"
Sep 15 19:07:26.086447 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:07:26.086429    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba\": container with ID starting with 82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba not found: ID does not exist" containerID="82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba"
Sep 15 19:07:26.086491 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.086453    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba"} err="failed to get container status \"82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba\": rpc error: code = NotFound desc = could not find container \"82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba\": container with ID starting with 82591cbe6213a6c71e514d4de813289388017c48350219ba48c722c56432efba not found: ID does not exist"
Sep 15 19:07:26.094810 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.094373    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-zhwmm"]
Sep 15 19:07:26.102206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:26.102178    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-zhwmm"]
Sep 15 19:07:27.677878 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:27.677845    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="251fc872-19bd-4d0f-a53d-81c2a7fbef12" path="/var/lib/kubelet/pods/251fc872-19bd-4d0f-a53d-81c2a7fbef12/volumes"
Sep 15 19:07:48.126476 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.126437    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"]
Sep 15 19:07:48.127022 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.126930    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="251fc872-19bd-4d0f-a53d-81c2a7fbef12" containerName="manager"
Sep 15 19:07:48.127022 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.126945    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="251fc872-19bd-4d0f-a53d-81c2a7fbef12" containerName="manager"
Sep 15 19:07:48.127022 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.127019    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="251fc872-19bd-4d0f-a53d-81c2a7fbef12" containerName="manager"
Sep 15 19:07:48.131887 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.131870    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.134158 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.134138    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9de44d-serving-cert\""
Sep 15 19:07:48.139936 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.139869    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"]
Sep 15 19:07:48.209239 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.209202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7679\" (UniqueName: \"kubernetes.io/projected/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-kube-api-access-z7679\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.209406 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.209267    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-metrics-certs\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.209406 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.209306    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-maas-api-tls\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.209406 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.209351    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.310214 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.310178    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7679\" (UniqueName: \"kubernetes.io/projected/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-kube-api-access-z7679\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.310403 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.310235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-metrics-certs\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.310403 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.310268    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-maas-api-tls\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.310403 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.310300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.311002 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.310977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.312929 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.312909    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-maas-api-tls\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.313092 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.312931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-metrics-certs\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.318513 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.318487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7679\" (UniqueName: \"kubernetes.io/projected/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-kube-api-access-z7679\") pod \"maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.443521 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.443424    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:48.720428 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.720404    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"]
Sep 15 19:07:48.722380 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:48.722354    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f4e1c0_3ca9_4c70_8283_1f83791d0bfa.slice/crio-1f8418a323e7189bce9f9e1a11e2676d7c8a7db2fe90b7b16dc4f8c7939e9e4d WatchSource:0}: Error finding container 1f8418a323e7189bce9f9e1a11e2676d7c8a7db2fe90b7b16dc4f8c7939e9e4d: Status 404 returned error can't find the container with id 1f8418a323e7189bce9f9e1a11e2676d7c8a7db2fe90b7b16dc4f8c7939e9e4d
Sep 15 19:07:48.723842 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.723822    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:07:48.868655 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.868602    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"]
Sep 15 19:07:48.873490 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.873470    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:48.881214 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.881109    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9de44d\""
Sep 15 19:07:48.881613 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.881504    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9de44d-dockercfg-2lfn6\""
Sep 15 19:07:48.885741 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:48.885714    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"]
Sep 15 19:07:49.018038 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.017950    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8847446-857a-4741-a03f-076891defe53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5\" (UID: \"f8847446-857a-4741-a03f-076891defe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:49.018038 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.018016    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmm7h\" (UniqueName: \"kubernetes.io/projected/f8847446-857a-4741-a03f-076891defe53-kube-api-access-jmm7h\") pod \"payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5\" (UID: \"f8847446-857a-4741-a03f-076891defe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:49.119446 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.119405    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8847446-857a-4741-a03f-076891defe53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5\" (UID: \"f8847446-857a-4741-a03f-076891defe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:49.119633 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.119503    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmm7h\" (UniqueName: \"kubernetes.io/projected/f8847446-857a-4741-a03f-076891defe53-kube-api-access-jmm7h\") pod \"payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5\" (UID: \"f8847446-857a-4741-a03f-076891defe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:49.120154 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.120136    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8847446-857a-4741-a03f-076891defe53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5\" (UID: \"f8847446-857a-4741-a03f-076891defe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:49.135596 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.135570    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmm7h\" (UniqueName: \"kubernetes.io/projected/f8847446-857a-4741-a03f-076891defe53-kube-api-access-jmm7h\") pod \"payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5\" (UID: \"f8847446-857a-4741-a03f-076891defe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:49.191727 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.191701    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:49.200798 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.200772    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf" event={"ID":"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa","Type":"ContainerStarted","Data":"1f8418a323e7189bce9f9e1a11e2676d7c8a7db2fe90b7b16dc4f8c7939e9e4d"}
Sep 15 19:07:49.465453 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.465126    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"]
Sep 15 19:07:49.467985 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:49.467957    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8847446_857a_4741_a03f_076891defe53.slice/crio-7e572d69bdba5d3d4fd329c9da65b0979b86b52b6c230a2ca015c1ae29636362 WatchSource:0}: Error finding container 7e572d69bdba5d3d4fd329c9da65b0979b86b52b6c230a2ca015c1ae29636362: Status 404 returned error can't find the container with id 7e572d69bdba5d3d4fd329c9da65b0979b86b52b6c230a2ca015c1ae29636362
Sep 15 19:07:49.570285 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.570246    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9011-5d795f"]
Sep 15 19:07:49.573848 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.573830    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-5d795f"
Sep 15 19:07:49.575991 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.575973    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgkcn\""
Sep 15 19:07:49.581972 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.581919    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9011-5d795f"]
Sep 15 19:07:49.725169 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.725085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhc5x\" (UniqueName: \"kubernetes.io/projected/77810fca-eebd-4b75-b64e-36b142399e47-kube-api-access-bhc5x\") pod \"test-curl-9011-5d795f\" (UID: \"77810fca-eebd-4b75-b64e-36b142399e47\") " pod="opendatahub/test-curl-9011-5d795f"
Sep 15 19:07:49.738440 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.738407    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"]
Sep 15 19:07:49.742100 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.742080    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:49.744620 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.744601    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f0ad35\""
Sep 15 19:07:49.744830 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.744812    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f0ad35-dockercfg-mvmcr\""
Sep 15 19:07:49.755338 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.755314    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"]
Sep 15 19:07:49.826768 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.826726    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv7hh\" (UniqueName: \"kubernetes.io/projected/6665de43-ff3f-4afa-a26b-899f208c7723-kube-api-access-bv7hh\") pod \"payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt\" (UID: \"6665de43-ff3f-4afa-a26b-899f208c7723\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:49.826959 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.826789    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhc5x\" (UniqueName: \"kubernetes.io/projected/77810fca-eebd-4b75-b64e-36b142399e47-kube-api-access-bhc5x\") pod \"test-curl-9011-5d795f\" (UID: \"77810fca-eebd-4b75-b64e-36b142399e47\") " pod="opendatahub/test-curl-9011-5d795f"
Sep 15 19:07:49.826959 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.826877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665de43-ff3f-4afa-a26b-899f208c7723-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt\" (UID: \"6665de43-ff3f-4afa-a26b-899f208c7723\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:49.836060 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.836032    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhc5x\" (UniqueName: \"kubernetes.io/projected/77810fca-eebd-4b75-b64e-36b142399e47-kube-api-access-bhc5x\") pod \"test-curl-9011-5d795f\" (UID: \"77810fca-eebd-4b75-b64e-36b142399e47\") " pod="opendatahub/test-curl-9011-5d795f"
Sep 15 19:07:49.888532 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.888499    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-5d795f"
Sep 15 19:07:49.928213 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.928174    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665de43-ff3f-4afa-a26b-899f208c7723-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt\" (UID: \"6665de43-ff3f-4afa-a26b-899f208c7723\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:49.928365 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.928290    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv7hh\" (UniqueName: \"kubernetes.io/projected/6665de43-ff3f-4afa-a26b-899f208c7723-kube-api-access-bv7hh\") pod \"payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt\" (UID: \"6665de43-ff3f-4afa-a26b-899f208c7723\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:49.929025 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.928979    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665de43-ff3f-4afa-a26b-899f208c7723-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt\" (UID: \"6665de43-ff3f-4afa-a26b-899f208c7723\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:49.942629 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:49.942601    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv7hh\" (UniqueName: \"kubernetes.io/projected/6665de43-ff3f-4afa-a26b-899f208c7723-kube-api-access-bv7hh\") pod \"payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt\" (UID: \"6665de43-ff3f-4afa-a26b-899f208c7723\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:50.056017 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.055926    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:50.163057 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.162985    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9011-5d795f"]
Sep 15 19:07:50.166424 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:50.166398    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77810fca_eebd_4b75_b64e_36b142399e47.slice/crio-4823c97997052d18b01d361c2b2a92f32e687a3a8b9f9a6e11e73580fd4a3247 WatchSource:0}: Error finding container 4823c97997052d18b01d361c2b2a92f32e687a3a8b9f9a6e11e73580fd4a3247: Status 404 returned error can't find the container with id 4823c97997052d18b01d361c2b2a92f32e687a3a8b9f9a6e11e73580fd4a3247
Sep 15 19:07:50.189310 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.189279    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"]
Sep 15 19:07:50.194499 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.194481    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.197036 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.197019    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-900c88-serving-cert\""
Sep 15 19:07:50.207239 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.207213    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"]
Sep 15 19:07:50.211723 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.211683    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9011-5d795f" event={"ID":"77810fca-eebd-4b75-b64e-36b142399e47","Type":"ContainerStarted","Data":"4823c97997052d18b01d361c2b2a92f32e687a3a8b9f9a6e11e73580fd4a3247"}
Sep 15 19:07:50.214081 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.214056    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf" event={"ID":"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa","Type":"ContainerStarted","Data":"8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33"}
Sep 15 19:07:50.214267 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.214246    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:50.216446 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.216422    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5" event={"ID":"f8847446-857a-4741-a03f-076891defe53","Type":"ContainerStarted","Data":"2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea"}
Sep 15 19:07:50.216538 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.216467    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5" event={"ID":"f8847446-857a-4741-a03f-076891defe53","Type":"ContainerStarted","Data":"7e572d69bdba5d3d4fd329c9da65b0979b86b52b6c230a2ca015c1ae29636362"}
Sep 15 19:07:50.216538 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.216507    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:07:50.235381 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.235336    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf" podStartSLOduration=1.067066604 podStartE2EDuration="2.23532182s" totalImagesPullingTime="1.168255216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:50.234270296 +0000 UTC m=+1029.090123567" watchObservedRunningTime="2026-09-15 19:07:50.23532182 +0000 UTC m=+1029.091175088"
Sep 15 19:07:50.250665 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.250613    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5" podStartSLOduration=2.250598538 podStartE2EDuration="2.250598538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:50.250456461 +0000 UTC m=+1029.106309731" watchObservedRunningTime="2026-09-15 19:07:50.250598538 +0000 UTC m=+1029.106451805"
Sep 15 19:07:50.333205 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.333170    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"]
Sep 15 19:07:50.333330 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.333306    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-metrics-certs\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.333803 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.333468    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcaeca-caa2-46f1-8b42-922e878ed390-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.333803 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.333557    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kxph\" (UniqueName: \"kubernetes.io/projected/83fcaeca-caa2-46f1-8b42-922e878ed390-kube-api-access-9kxph\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.333957 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.333908    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.334683 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:50.334632    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6665de43_ff3f_4afa_a26b_899f208c7723.slice/crio-7c41f0bbd687428f2e0661d0482927db14f42187c3bab76543eb877cd597b8b8 WatchSource:0}: Error finding container 7c41f0bbd687428f2e0661d0482927db14f42187c3bab76543eb877cd597b8b8: Status 404 returned error can't find the container with id 7c41f0bbd687428f2e0661d0482927db14f42187c3bab76543eb877cd597b8b8
Sep 15 19:07:50.435001 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.434969    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kxph\" (UniqueName: \"kubernetes.io/projected/83fcaeca-caa2-46f1-8b42-922e878ed390-kube-api-access-9kxph\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.435138 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.435020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.435138 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.435061    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-metrics-certs\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.435138 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.435122    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcaeca-caa2-46f1-8b42-922e878ed390-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.435312 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:07:50.435203    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-serving-cert: secret "maas-api-e2e-worker-w0-900c88-serving-cert" not found
Sep 15 19:07:50.435312 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:07:50.435280    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls podName:83fcaeca-caa2-46f1-8b42-922e878ed390 nodeName:}" failed. No retries permitted until 2026-09-15 19:07:50.935255279 +0000 UTC m=+1029.791108537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls") pod "maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg" (UID: "83fcaeca-caa2-46f1-8b42-922e878ed390") : secret "maas-api-e2e-worker-w0-900c88-serving-cert" not found
Sep 15 19:07:50.435825 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.435802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcaeca-caa2-46f1-8b42-922e878ed390-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.437443 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.437421    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-metrics-certs\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.445519 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.445496    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kxph\" (UniqueName: \"kubernetes.io/projected/83fcaeca-caa2-46f1-8b42-922e878ed390-kube-api-access-9kxph\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.940074 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.939956    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:50.942951 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:50.942924    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls\") pod \"maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:51.111229 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:51.111189    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:51.227569 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:51.227480    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt" event={"ID":"6665de43-ff3f-4afa-a26b-899f208c7723","Type":"ContainerStarted","Data":"7ecd665a3be629a4f4fc2df52d7de55e0adc7ccf5bf4fb983768a1bcb361e799"}
Sep 15 19:07:51.227569 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:51.227552    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt" event={"ID":"6665de43-ff3f-4afa-a26b-899f208c7723","Type":"ContainerStarted","Data":"7c41f0bbd687428f2e0661d0482927db14f42187c3bab76543eb877cd597b8b8"}
Sep 15 19:07:51.228186 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:51.228165    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:07:51.243269 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:51.243216    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt" podStartSLOduration=2.243196609 podStartE2EDuration="2.243196609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:51.242211063 +0000 UTC m=+1030.098064331" watchObservedRunningTime="2026-09-15 19:07:51.243196609 +0000 UTC m=+1030.099049878"
Sep 15 19:07:51.394584 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:51.394507    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"]
Sep 15 19:07:52.126631 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:52.126590    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fcaeca_caa2_46f1_8b42_922e878ed390.slice/crio-58d71fb5c7eea04735405b867f77c3959f244f41ce147b0aa35d283ebf234744 WatchSource:0}: Error finding container 58d71fb5c7eea04735405b867f77c3959f244f41ce147b0aa35d283ebf234744: Status 404 returned error can't find the container with id 58d71fb5c7eea04735405b867f77c3959f244f41ce147b0aa35d283ebf234744
Sep 15 19:07:52.234577 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:52.234548    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg" event={"ID":"83fcaeca-caa2-46f1-8b42-922e878ed390","Type":"ContainerStarted","Data":"58d71fb5c7eea04735405b867f77c3959f244f41ce147b0aa35d283ebf234744"}
Sep 15 19:07:53.242516 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:53.242472    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9011-5d795f" event={"ID":"77810fca-eebd-4b75-b64e-36b142399e47","Type":"ContainerStarted","Data":"79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396"}
Sep 15 19:07:53.244752 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:53.244723    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg" event={"ID":"83fcaeca-caa2-46f1-8b42-922e878ed390","Type":"ContainerStarted","Data":"48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc"}
Sep 15 19:07:53.245046 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:53.245022    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:53.416994 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:53.416938    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg" podStartSLOduration=2.945746622 podStartE2EDuration="3.416920917s" totalImagesPullingTime="471.174295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:53.414917659 +0000 UTC m=+1032.270770929" watchObservedRunningTime="2026-09-15 19:07:53.416920917 +0000 UTC m=+1032.272774184"
Sep 15 19:07:53.418332 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:53.418300    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9011-5d795f" podStartSLOduration=2.38391382 podStartE2EDuration="4.418288963s" totalImagesPullingTime="2.034375143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:53.357915575 +0000 UTC m=+1032.213768842" watchObservedRunningTime="2026-09-15 19:07:53.418288963 +0000 UTC m=+1032.274142233"
Sep 15 19:07:53.982428 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:53.982388    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9011-5d795f"]
Sep 15 19:07:53.988207 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:53.988175    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9011-5d795f"]
Sep 15 19:07:54.250725 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:07:54.250627    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9011-5d795f" container="test-curl-9011-5d795f"
Sep 15 19:07:55.256515 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:55.256473    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9011-5d795f" podUID="77810fca-eebd-4b75-b64e-36b142399e47" containerName="test-curl-9011-5d795f" containerID="cri-o://79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396" gracePeriod=2
Sep 15 19:07:55.258931 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:07:55.258902    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9011-5d795f\" is forbidden: User \"system:node:ip-10-0-140-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-183.ec2.internal' and this object" podUID="77810fca-eebd-4b75-b64e-36b142399e47" pod="opendatahub/test-curl-9011-5d795f"
Sep 15 19:07:56.233677 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:56.233632    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:07:57.100339 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.100282    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"]
Sep 15 19:07:57.100910 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.100889    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77810fca-eebd-4b75-b64e-36b142399e47" containerName="test-curl-9011-5d795f"
Sep 15 19:07:57.100956 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.100912    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="77810fca-eebd-4b75-b64e-36b142399e47" containerName="test-curl-9011-5d795f"
Sep 15 19:07:57.100989 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.100980    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="77810fca-eebd-4b75-b64e-36b142399e47" containerName="test-curl-9011-5d795f"
Sep 15 19:07:57.104369 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.104351    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.106669 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.106633    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a7a10c67-serving-cert\""
Sep 15 19:07:57.114249 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.114223    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"]
Sep 15 19:07:57.205982 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.205950    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.206154 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.206022    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh54q\" (UniqueName: \"kubernetes.io/projected/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-kube-api-access-dh54q\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.206154 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.206076    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-metrics-certs\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.206244 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.206155    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-openshift-service-ca\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.306709 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.306685    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.306814 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.306758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh54q\" (UniqueName: \"kubernetes.io/projected/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-kube-api-access-dh54q\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.306814 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.306808    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-metrics-certs\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.306906 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.306835    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-openshift-service-ca\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.306906 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:07:57.306849    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-serving-cert: secret "maas-api-e2e-ait-a7a10c67-serving-cert" not found
Sep 15 19:07:57.306977 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:07:57.306922    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls podName:d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39 nodeName:}" failed. No retries permitted until 2026-09-15 19:07:57.806902131 +0000 UTC m=+1036.662755401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls") pod "maas-api-e2e-ait-a7a10c67-7db45f948b-css8g" (UID: "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39") : secret "maas-api-e2e-ait-a7a10c67-serving-cert" not found
Sep 15 19:07:57.307416 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.307395    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-openshift-service-ca\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.309122 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.309103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-metrics-certs\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.314879 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.314853    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh54q\" (UniqueName: \"kubernetes.io/projected/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-kube-api-access-dh54q\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.432688 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.432667    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-5d795f"
Sep 15 19:07:57.609605 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.609577    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhc5x\" (UniqueName: \"kubernetes.io/projected/77810fca-eebd-4b75-b64e-36b142399e47-kube-api-access-bhc5x\") pod \"77810fca-eebd-4b75-b64e-36b142399e47\" (UID: \"77810fca-eebd-4b75-b64e-36b142399e47\") "
Sep 15 19:07:57.611571 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.611549    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77810fca-eebd-4b75-b64e-36b142399e47-kube-api-access-bhc5x" (OuterVolumeSpecName: "kube-api-access-bhc5x") pod "77810fca-eebd-4b75-b64e-36b142399e47" (UID: "77810fca-eebd-4b75-b64e-36b142399e47"). InnerVolumeSpecName "kube-api-access-bhc5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:07:57.679672 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.679610    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77810fca-eebd-4b75-b64e-36b142399e47" path="/var/lib/kubelet/pods/77810fca-eebd-4b75-b64e-36b142399e47/volumes"
Sep 15 19:07:57.711149 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.711116    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhc5x\" (UniqueName: \"kubernetes.io/projected/77810fca-eebd-4b75-b64e-36b142399e47-kube-api-access-bhc5x\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:07:57.786397 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.786369    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"]
Sep 15 19:07:57.790153 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.790134    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:57.792509 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.792487    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a7a10c67\""
Sep 15 19:07:57.792590 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.792539    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a7a10c67-dockercfg-z9h9d\""
Sep 15 19:07:57.798532 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.798507    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"]
Sep 15 19:07:57.812033 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.812011    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.814550 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.814528    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls\") pod \"maas-api-e2e-ait-a7a10c67-7db45f948b-css8g\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:57.913542 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.913507    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6815ec-87e3-44bf-a33b-6df9353d06c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk\" (UID: \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:57.913701 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:57.913590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsck4\" (UniqueName: \"kubernetes.io/projected/cc6815ec-87e3-44bf-a33b-6df9353d06c3-kube-api-access-gsck4\") pod \"payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk\" (UID: \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:58.015092 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.015020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsck4\" (UniqueName: \"kubernetes.io/projected/cc6815ec-87e3-44bf-a33b-6df9353d06c3-kube-api-access-gsck4\") pod \"payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk\" (UID: \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:58.015218 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.015159    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6815ec-87e3-44bf-a33b-6df9353d06c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk\" (UID: \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:58.015867 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.015847    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6815ec-87e3-44bf-a33b-6df9353d06c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk\" (UID: \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:58.017928 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.017912    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:58.022782 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.022763    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsck4\" (UniqueName: \"kubernetes.io/projected/cc6815ec-87e3-44bf-a33b-6df9353d06c3-kube-api-access-gsck4\") pod \"payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk\" (UID: \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:58.102056 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.102023    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:58.275558 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.275477    2556 generic.go:356] "Generic (PLEG): container finished" podID="77810fca-eebd-4b75-b64e-36b142399e47" containerID="79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396" exitCode=137
Sep 15 19:07:58.275558 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.275518    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-5d795f"
Sep 15 19:07:58.275757 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.275564    2556 scope.go:117] "RemoveContainer" containerID="79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396"
Sep 15 19:07:58.284772 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.284754    2556 scope.go:117] "RemoveContainer" containerID="79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396"
Sep 15 19:07:58.285033 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:07:58.285015    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396\": container with ID starting with 79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396 not found: ID does not exist" containerID="79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396"
Sep 15 19:07:58.285084 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.285043    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396"} err="failed to get container status \"79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396\": rpc error: code = NotFound desc = could not find container \"79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396\": container with ID starting with 79afa136e8f33c3e221633964d974058a1576e5e666eb66bc4bc41102b92a396 not found: ID does not exist"
Sep 15 19:07:58.290310 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.290289    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"]
Sep 15 19:07:58.292591 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:58.292572    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50bf0f4_81ff_4a5b_8ecb_a25f3f0a7d39.slice/crio-723db33689792e1922fe194a9c22c0d04f963734d8afb4230faf1dbb0d5b437d WatchSource:0}: Error finding container 723db33689792e1922fe194a9c22c0d04f963734d8afb4230faf1dbb0d5b437d: Status 404 returned error can't find the container with id 723db33689792e1922fe194a9c22c0d04f963734d8afb4230faf1dbb0d5b437d
Sep 15 19:07:58.373009 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:58.372988    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"]
Sep 15 19:07:58.374810 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:07:58.374785    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6815ec_87e3_44bf_a33b_6df9353d06c3.slice/crio-c3c88b8b8012e23c467777a73278cc40eea0a4c5de27f492685add757a97ceef WatchSource:0}: Error finding container c3c88b8b8012e23c467777a73278cc40eea0a4c5de27f492685add757a97ceef: Status 404 returned error can't find the container with id c3c88b8b8012e23c467777a73278cc40eea0a4c5de27f492685add757a97ceef
Sep 15 19:07:59.256616 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.256583    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:07:59.283297 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.283263    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g" event={"ID":"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39","Type":"ContainerStarted","Data":"4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59"}
Sep 15 19:07:59.283483 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.283321    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g" event={"ID":"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39","Type":"ContainerStarted","Data":"723db33689792e1922fe194a9c22c0d04f963734d8afb4230faf1dbb0d5b437d"}
Sep 15 19:07:59.283483 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.283382    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:07:59.286780 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.286752    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk" event={"ID":"cc6815ec-87e3-44bf-a33b-6df9353d06c3","Type":"ContainerStarted","Data":"2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796"}
Sep 15 19:07:59.286934 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.286803    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk" event={"ID":"cc6815ec-87e3-44bf-a33b-6df9353d06c3","Type":"ContainerStarted","Data":"c3c88b8b8012e23c467777a73278cc40eea0a4c5de27f492685add757a97ceef"}
Sep 15 19:07:59.286934 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.286834    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:07:59.299721 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.299523    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g" podStartSLOduration=1.7306891100000001 podStartE2EDuration="2.299504633s" totalImagesPullingTime="568.815523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:59.298558517 +0000 UTC m=+1038.154411799" watchObservedRunningTime="2026-09-15 19:07:59.299504633 +0000 UTC m=+1038.155357902"
Sep 15 19:07:59.313996 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:07:59.313949    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk" podStartSLOduration=2.313930254 podStartE2EDuration="2.313930254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:59.312128132 +0000 UTC m=+1038.167981414" watchObservedRunningTime="2026-09-15 19:07:59.313930254 +0000 UTC m=+1038.169783522"
Sep 15 19:08:01.230575 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:01.230546    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:08:03.247481 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:03.247452    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:08:05.298304 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:05.298278    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:08:10.295765 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:10.295733    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:08:17.569582 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.569535    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"]
Sep 15 19:08:17.573596 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.573579    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.575915 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.575896    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"kube-root-ca.crt\""
Sep 15 19:08:17.576953 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.576927    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"default-dockercfg-cv7tx\""
Sep 15 19:08:17.577053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.576949    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"openshift-service-ca.crt\""
Sep 15 19:08:17.577053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.576988    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"simulator-w3-f0ad35-kserve-self-signed-certs\""
Sep 15 19:08:17.584030 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.584006    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"]
Sep 15 19:08:17.707366 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.707332    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-home\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.707366 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.707376    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-dshm\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.707589 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.707398    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tls-certs\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.707589 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.707447    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tmp-dir\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.707589 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.707491    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzdk\" (UniqueName: \"kubernetes.io/projected/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-kube-api-access-jhzdk\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.707589 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.707558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-model-cache\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.808270 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.808235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzdk\" (UniqueName: \"kubernetes.io/projected/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-kube-api-access-jhzdk\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.808475 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.808452    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-model-cache\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.808573 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.808501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-home\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.808573 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.808547    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-dshm\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.808740 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.808610    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tls-certs\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.808806 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.808760    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tmp-dir\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.809223 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.809198    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-home\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.809371 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.809309    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-model-cache\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.809511 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.809483    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tmp-dir\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.810981 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.810960    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-dshm\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.811317 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.811298    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tls-certs\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.819501 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.819476    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzdk\" (UniqueName: \"kubernetes.io/projected/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-kube-api-access-jhzdk\") pod \"simulator-w3-f0ad35-kserve-67d549dd85-bw4jz\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") " pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:17.884312 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:17.884285    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:18.175011 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.174822    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"]
Sep 15 19:08:18.177124 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:08:18.177097    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bf02fcb_e9dc_448a_a3d5_b3f511de7b1e.slice/crio-d49587eddf6da11968ac022e6a74a0d5c2f1b085702fc477053dfa730cd58f20 WatchSource:0}: Error finding container d49587eddf6da11968ac022e6a74a0d5c2f1b085702fc477053dfa730cd58f20: Status 404 returned error can't find the container with id d49587eddf6da11968ac022e6a74a0d5c2f1b085702fc477053dfa730cd58f20
Sep 15 19:08:18.267038 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.267001    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"]
Sep 15 19:08:18.271918 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.271895    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.274539 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.274518    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3daa58-serving-cert\""
Sep 15 19:08:18.288290 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.288269    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"]
Sep 15 19:08:18.399264 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.399201    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz" event={"ID":"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e","Type":"ContainerStarted","Data":"add4fed4b0b21bff2cf6b1b203b2167839bc9c43dfb1e08e3b90ae45992a1b21"}
Sep 15 19:08:18.399445 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.399269    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz" event={"ID":"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e","Type":"ContainerStarted","Data":"d49587eddf6da11968ac022e6a74a0d5c2f1b085702fc477053dfa730cd58f20"}
Sep 15 19:08:18.414808 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.414781    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c42kh\" (UniqueName: \"kubernetes.io/projected/71be7022-f1b9-478c-8035-1d4ab85a0d7c-kube-api-access-c42kh\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.414942 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.414817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.414942 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.414851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be7022-f1b9-478c-8035-1d4ab85a0d7c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.414942 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.414926    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-metrics-certs\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.419783 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.419737    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz" podStartSLOduration=1.4197237870000001 podStartE2EDuration="1.419723787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:18.418256497 +0000 UTC m=+1057.274109765" watchObservedRunningTime="2026-09-15 19:08:18.419723787 +0000 UTC m=+1057.275577054"
Sep 15 19:08:18.516039 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.515957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c42kh\" (UniqueName: \"kubernetes.io/projected/71be7022-f1b9-478c-8035-1d4ab85a0d7c-kube-api-access-c42kh\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.516039 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.516001    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.516245 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.516044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be7022-f1b9-478c-8035-1d4ab85a0d7c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.516245 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.516087    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-metrics-certs\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.516245 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:08:18.516180    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-serving-cert: secret "maas-api-e2e-inf-b-3daa58-serving-cert" not found
Sep 15 19:08:18.516407 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:08:18.516264    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls podName:71be7022-f1b9-478c-8035-1d4ab85a0d7c nodeName:}" failed. No retries permitted until 2026-09-15 19:08:19.0162431 +0000 UTC m=+1057.872096367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls") pod "maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz" (UID: "71be7022-f1b9-478c-8035-1d4ab85a0d7c") : secret "maas-api-e2e-inf-b-3daa58-serving-cert" not found
Sep 15 19:08:18.516752 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.516731    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be7022-f1b9-478c-8035-1d4ab85a0d7c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.518551 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.518531    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-metrics-certs\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:18.530993 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:18.530969    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c42kh\" (UniqueName: \"kubernetes.io/projected/71be7022-f1b9-478c-8035-1d4ab85a0d7c-kube-api-access-c42kh\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:19.020393 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:19.020357    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:19.023069 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:19.023042    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls\") pod \"maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:19.183145 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:19.183112    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:19.467730 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:19.467701    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"]
Sep 15 19:08:19.470922 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:08:19.470898    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71be7022_f1b9_478c_8035_1d4ab85a0d7c.slice/crio-007d5c5b6fd681138de64efe8d086c26dbebb0e43b68ec35072574b9d17056e7 WatchSource:0}: Error finding container 007d5c5b6fd681138de64efe8d086c26dbebb0e43b68ec35072574b9d17056e7: Status 404 returned error can't find the container with id 007d5c5b6fd681138de64efe8d086c26dbebb0e43b68ec35072574b9d17056e7
Sep 15 19:08:19.671912 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:19.671882    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:08:20.413071 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:20.412983    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz" event={"ID":"71be7022-f1b9-478c-8035-1d4ab85a0d7c","Type":"ContainerStarted","Data":"a2d6bf42e5f79222497ce722b65113912d665cf8138c8461f8c592623a020ac4"}
Sep 15 19:08:20.413071 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:20.413049    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:20.413071 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:20.413068    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz" event={"ID":"71be7022-f1b9-478c-8035-1d4ab85a0d7c","Type":"ContainerStarted","Data":"007d5c5b6fd681138de64efe8d086c26dbebb0e43b68ec35072574b9d17056e7"}
Sep 15 19:08:20.428649 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:20.428603    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz" podStartSLOduration=1.739087802 podStartE2EDuration="2.42858638s" totalImagesPullingTime="689.498578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:20.428237849 +0000 UTC m=+1059.284091117" watchObservedRunningTime="2026-09-15 19:08:20.42858638 +0000 UTC m=+1059.284439647"
Sep 15 19:08:25.487174 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.487127    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"]
Sep 15 19:08:25.487625 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.487397    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g" podUID="d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" containerName="maas-api" containerID="cri-o://4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59" gracePeriod=30
Sep 15 19:08:25.535848 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.535816    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"]
Sep 15 19:08:25.536077 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.536031    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk" podUID="cc6815ec-87e3-44bf-a33b-6df9353d06c3" containerName="payload-pre-processing" containerID="cri-o://2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796" gracePeriod=30
Sep 15 19:08:25.796507 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.796479    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:08:25.832565 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.832511    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:08:25.894264 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.894231    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh54q\" (UniqueName: \"kubernetes.io/projected/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-kube-api-access-dh54q\") pod \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") "
Sep 15 19:08:25.894264 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.894266    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-metrics-certs\") pod \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") "
Sep 15 19:08:25.894449 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.894395    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-openshift-service-ca\") pod \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") "
Sep 15 19:08:25.894494 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.894460    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls\") pod \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\" (UID: \"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39\") "
Sep 15 19:08:25.894755 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.894720    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" (UID: "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:08:25.894918 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.894903    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-openshift-service-ca\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:08:25.896582 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.896562    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" (UID: "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:08:25.896889 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.896865    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" (UID: "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:08:25.896942 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.896870    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-kube-api-access-dh54q" (OuterVolumeSpecName: "kube-api-access-dh54q") pod "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" (UID: "d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39"). InnerVolumeSpecName "kube-api-access-dh54q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:08:25.995755 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.995687    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6815ec-87e3-44bf-a33b-6df9353d06c3-plugins-config-volume\") pod \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\" (UID: \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\") "
Sep 15 19:08:25.995882 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.995783    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsck4\" (UniqueName: \"kubernetes.io/projected/cc6815ec-87e3-44bf-a33b-6df9353d06c3-kube-api-access-gsck4\") pod \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\" (UID: \"cc6815ec-87e3-44bf-a33b-6df9353d06c3\") "
Sep 15 19:08:25.996039 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.996014    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6815ec-87e3-44bf-a33b-6df9353d06c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc6815ec-87e3-44bf-a33b-6df9353d06c3" (UID: "cc6815ec-87e3-44bf-a33b-6df9353d06c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:08:25.996082 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.996072    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh54q\" (UniqueName: \"kubernetes.io/projected/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-kube-api-access-dh54q\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:08:25.996117 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.996090    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:08:25.996117 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.996103    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39-maas-api-tls\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:08:25.997691 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:25.997669    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6815ec-87e3-44bf-a33b-6df9353d06c3-kube-api-access-gsck4" (OuterVolumeSpecName: "kube-api-access-gsck4") pod "cc6815ec-87e3-44bf-a33b-6df9353d06c3" (UID: "cc6815ec-87e3-44bf-a33b-6df9353d06c3"). InnerVolumeSpecName "kube-api-access-gsck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:08:26.097843 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.097809    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsck4\" (UniqueName: \"kubernetes.io/projected/cc6815ec-87e3-44bf-a33b-6df9353d06c3-kube-api-access-gsck4\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:08:26.098013 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.097957    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6815ec-87e3-44bf-a33b-6df9353d06c3-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:08:26.424734 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.424702    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:08:26.450061 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.450030    2556 generic.go:356] "Generic (PLEG): container finished" podID="d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" containerID="4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59" exitCode=0
Sep 15 19:08:26.450217 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.450073    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"
Sep 15 19:08:26.450217 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.450115    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g" event={"ID":"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39","Type":"ContainerDied","Data":"4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59"}
Sep 15 19:08:26.450217 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.450165    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g" event={"ID":"d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39","Type":"ContainerDied","Data":"723db33689792e1922fe194a9c22c0d04f963734d8afb4230faf1dbb0d5b437d"}
Sep 15 19:08:26.450217 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.450200    2556 scope.go:117] "RemoveContainer" containerID="4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59"
Sep 15 19:08:26.451835 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.451810    2556 generic.go:356] "Generic (PLEG): container finished" podID="cc6815ec-87e3-44bf-a33b-6df9353d06c3" containerID="2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796" exitCode=0
Sep 15 19:08:26.451937 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.451876    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"
Sep 15 19:08:26.451937 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.451918    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk" event={"ID":"cc6815ec-87e3-44bf-a33b-6df9353d06c3","Type":"ContainerDied","Data":"2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796"}
Sep 15 19:08:26.452043 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.451982    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk" event={"ID":"cc6815ec-87e3-44bf-a33b-6df9353d06c3","Type":"ContainerDied","Data":"c3c88b8b8012e23c467777a73278cc40eea0a4c5de27f492685add757a97ceef"}
Sep 15 19:08:26.462265 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.462243    2556 scope.go:117] "RemoveContainer" containerID="4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59"
Sep 15 19:08:26.463018 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:08:26.462987    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59\": container with ID starting with 4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59 not found: ID does not exist" containerID="4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59"
Sep 15 19:08:26.463124 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.463027    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59"} err="failed to get container status \"4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59\": rpc error: code = NotFound desc = could not find container \"4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59\": container with ID starting with 4cf18545cc7d35f1c914517ae3596312f90220d5548ca8f0d1d8969da7766e59 not found: ID does not exist"
Sep 15 19:08:26.463124 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.463052    2556 scope.go:117] "RemoveContainer" containerID="2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796"
Sep 15 19:08:26.473235 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.473216    2556 scope.go:117] "RemoveContainer" containerID="2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796"
Sep 15 19:08:26.473591 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:08:26.473568    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796\": container with ID starting with 2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796 not found: ID does not exist" containerID="2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796"
Sep 15 19:08:26.473724 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.473599    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796"} err="failed to get container status \"2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796\": rpc error: code = NotFound desc = could not find container \"2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796\": container with ID starting with 2fc72ff41127d9327ac5d394888e8b2383bbba441461b591986ac276ad08e796 not found: ID does not exist"
Sep 15 19:08:26.491400 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.491374    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"]
Sep 15 19:08:26.497495 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.497474    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7a10c67-7db45f948b-css8g"]
Sep 15 19:08:26.517149 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.517085    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"]
Sep 15 19:08:26.520046 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:26.520020    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7a10c67-65d478bb6b-4s6nk"]
Sep 15 19:08:27.678955 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:27.678917    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6815ec-87e3-44bf-a33b-6df9353d06c3" path="/var/lib/kubelet/pods/cc6815ec-87e3-44bf-a33b-6df9353d06c3/volumes"
Sep 15 19:08:27.679403 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:27.679386    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" path="/var/lib/kubelet/pods/d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39/volumes"
Sep 15 19:08:27.885384 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:27.885334    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:27.885384 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:27.885391    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:27.898301 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:27.898277    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:28.477425 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:28.477387    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:08:30.845995 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.845955    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"]
Sep 15 19:08:30.846910 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.846884    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" containerName="maas-api"
Sep 15 19:08:30.846910 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.846908    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" containerName="maas-api"
Sep 15 19:08:30.847111 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.846922    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6815ec-87e3-44bf-a33b-6df9353d06c3" containerName="payload-pre-processing"
Sep 15 19:08:30.847111 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.846931    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6815ec-87e3-44bf-a33b-6df9353d06c3" containerName="payload-pre-processing"
Sep 15 19:08:30.847111 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.847043    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50bf0f4-81ff-4a5b-8ecb-a25f3f0a7d39" containerName="maas-api"
Sep 15 19:08:30.847111 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.847060    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6815ec-87e3-44bf-a33b-6df9353d06c3" containerName="payload-pre-processing"
Sep 15 19:08:30.851817 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.851799    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:30.854172 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.854151    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8b287f5b\""
Sep 15 19:08:30.854172 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.854166    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8b287f5b-dockercfg-tkxr6\""
Sep 15 19:08:30.862584 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:30.862558    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"]
Sep 15 19:08:31.050380 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.050338    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg\" (UID: \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:31.050549 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.050425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtgg\" (UniqueName: \"kubernetes.io/projected/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-kube-api-access-jhtgg\") pod \"payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg\" (UID: \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:31.151609 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.151560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg\" (UID: \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:31.151837 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.151623    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtgg\" (UniqueName: \"kubernetes.io/projected/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-kube-api-access-jhtgg\") pod \"payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg\" (UID: \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:31.152297 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.152271    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg\" (UID: \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:31.159086 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.159061    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtgg\" (UniqueName: \"kubernetes.io/projected/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-kube-api-access-jhtgg\") pod \"payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg\" (UID: \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:31.165866 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.165846    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:31.441618 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.441588    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"]
Sep 15 19:08:31.443814 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:08:31.443788    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a78b8cc_b0a5_4561_89a7_14ff7f9fdc28.slice/crio-2bb1cefe0f006bedabd06d6a39bdb8697ab32c80417835b4e2b7f9d1c68321b9 WatchSource:0}: Error finding container 2bb1cefe0f006bedabd06d6a39bdb8697ab32c80417835b4e2b7f9d1c68321b9: Status 404 returned error can't find the container with id 2bb1cefe0f006bedabd06d6a39bdb8697ab32c80417835b4e2b7f9d1c68321b9
Sep 15 19:08:31.483495 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:31.483471    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg" event={"ID":"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28","Type":"ContainerStarted","Data":"2bb1cefe0f006bedabd06d6a39bdb8697ab32c80417835b4e2b7f9d1c68321b9"}
Sep 15 19:08:32.491984 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:32.491940    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg" event={"ID":"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28","Type":"ContainerStarted","Data":"a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06"}
Sep 15 19:08:32.493840 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:32.493808    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:08:32.511061 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:32.511005    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg" podStartSLOduration=2.510989056 podStartE2EDuration="2.510989056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:32.509233671 +0000 UTC m=+1071.365086980" watchObservedRunningTime="2026-09-15 19:08:32.510989056 +0000 UTC m=+1071.366842331"
Sep 15 19:08:34.672441 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:34.672371    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:08:44.505237 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:08:44.505203    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:09:01.691999 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:01.691964    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"]
Sep 15 19:09:01.692627 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:01.692157    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg" podUID="5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28" containerName="payload-processing" containerID="cri-o://a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06" gracePeriod=30
Sep 15 19:09:01.958784 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:01.958688    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:09:02.040498 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.040455    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhtgg\" (UniqueName: \"kubernetes.io/projected/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-kube-api-access-jhtgg\") pod \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\" (UID: \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\") "
Sep 15 19:09:02.040498 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.040503    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-plugins-config-volume\") pod \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\" (UID: \"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28\") "
Sep 15 19:09:02.040973 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.040947    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28" (UID: "5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:02.042761 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.042725    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-kube-api-access-jhtgg" (OuterVolumeSpecName: "kube-api-access-jhtgg") pod "5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28" (UID: "5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28"). InnerVolumeSpecName "kube-api-access-jhtgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:02.141397 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.141361    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtgg\" (UniqueName: \"kubernetes.io/projected/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-kube-api-access-jhtgg\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:09:02.141397 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.141392    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:09:02.661620 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.661579    2556 generic.go:356] "Generic (PLEG): container finished" podID="5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28" containerID="a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06" exitCode=0
Sep 15 19:09:02.661797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.661627    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"
Sep 15 19:09:02.661797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.661626    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg" event={"ID":"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28","Type":"ContainerDied","Data":"a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06"}
Sep 15 19:09:02.661797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.661707    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg" event={"ID":"5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28","Type":"ContainerDied","Data":"2bb1cefe0f006bedabd06d6a39bdb8697ab32c80417835b4e2b7f9d1c68321b9"}
Sep 15 19:09:02.661797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.661739    2556 scope.go:117] "RemoveContainer" containerID="a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06"
Sep 15 19:09:02.672234 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.672214    2556 scope.go:117] "RemoveContainer" containerID="a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06"
Sep 15 19:09:02.672506 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:09:02.672488    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06\": container with ID starting with a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06 not found: ID does not exist" containerID="a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06"
Sep 15 19:09:02.672587 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.672518    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06"} err="failed to get container status \"a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06\": rpc error: code = NotFound desc = could not find container \"a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06\": container with ID starting with a102bf5e97501032bdeb1a56e45f039dac7d02a667b16b7b96564709df33fb06 not found: ID does not exist"
Sep 15 19:09:02.689236 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.689206    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"]
Sep 15 19:09:02.696183 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:02.696156    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8b287f5b-6788947858-gjxwg"]
Sep 15 19:09:03.681617 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:03.681580    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28" path="/var/lib/kubelet/pods/5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28/volumes"
Sep 15 19:09:36.672318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:36.672281    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:09:41.676907 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:41.676103    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:09:45.241859 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.241827    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"]
Sep 15 19:09:45.242286 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.242272    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28" containerName="payload-processing"
Sep 15 19:09:45.242286 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.242287    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28" containerName="payload-processing"
Sep 15 19:09:45.242377 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.242367    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a78b8cc-b0a5-4561-89a7-14ff7f9fdc28" containerName="payload-processing"
Sep 15 19:09:45.246601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.246582    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.250107 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.250088    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-282a6587-dockercfg-ndzdt\""
Sep 15 19:09:45.250209 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.250088    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-282a6587\""
Sep 15 19:09:45.255188 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.255161    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"]
Sep 15 19:09:45.361051 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.361017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6e9629-982e-4da4-a70d-434e94785168-plugins-config-volume\") pod \"payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9\" (UID: \"5c6e9629-982e-4da4-a70d-434e94785168\") " pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.361198 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.361067    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8l2\" (UniqueName: \"kubernetes.io/projected/5c6e9629-982e-4da4-a70d-434e94785168-kube-api-access-zg8l2\") pod \"payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9\" (UID: \"5c6e9629-982e-4da4-a70d-434e94785168\") " pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.462489 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.462447    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6e9629-982e-4da4-a70d-434e94785168-plugins-config-volume\") pod \"payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9\" (UID: \"5c6e9629-982e-4da4-a70d-434e94785168\") " pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.462676 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.462498    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8l2\" (UniqueName: \"kubernetes.io/projected/5c6e9629-982e-4da4-a70d-434e94785168-kube-api-access-zg8l2\") pod \"payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9\" (UID: \"5c6e9629-982e-4da4-a70d-434e94785168\") " pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.463490 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.463466    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6e9629-982e-4da4-a70d-434e94785168-plugins-config-volume\") pod \"payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9\" (UID: \"5c6e9629-982e-4da4-a70d-434e94785168\") " pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.470448 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.470419    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8l2\" (UniqueName: \"kubernetes.io/projected/5c6e9629-982e-4da4-a70d-434e94785168-kube-api-access-zg8l2\") pod \"payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9\" (UID: \"5c6e9629-982e-4da4-a70d-434e94785168\") " pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.560247 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.560162    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.831814 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.831788    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"]
Sep 15 19:09:45.833918 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:09:45.833893    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6e9629_982e_4da4_a70d_434e94785168.slice/crio-b9af6da6a8554464c2173d0e4aad79327b4fd35ab60feed18c6ce2e43a605875 WatchSource:0}: Error finding container b9af6da6a8554464c2173d0e4aad79327b4fd35ab60feed18c6ce2e43a605875: Status 404 returned error can't find the container with id b9af6da6a8554464c2173d0e4aad79327b4fd35ab60feed18c6ce2e43a605875
Sep 15 19:09:45.900304 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.900277    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9" event={"ID":"5c6e9629-982e-4da4-a70d-434e94785168","Type":"ContainerStarted","Data":"b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0"}
Sep 15 19:09:45.900411 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.900328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9" event={"ID":"5c6e9629-982e-4da4-a70d-434e94785168","Type":"ContainerStarted","Data":"b9af6da6a8554464c2173d0e4aad79327b4fd35ab60feed18c6ce2e43a605875"}
Sep 15 19:09:45.900411 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.900381    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:09:45.917617 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:45.917560    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9" podStartSLOduration=0.917542967 podStartE2EDuration="917.542967ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:45.91479882 +0000 UTC m=+1144.770652083" watchObservedRunningTime="2026-09-15 19:09:45.917542967 +0000 UTC m=+1144.773396302"
Sep 15 19:09:56.907404 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:09:56.907371    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:10:23.277785 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.277577    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"]
Sep 15 19:10:23.278374 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.277852    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9" podUID="5c6e9629-982e-4da4-a70d-434e94785168" containerName="payload-processing" containerID="cri-o://b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0" gracePeriod=30
Sep 15 19:10:23.530158 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.530098    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:10:23.627356 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.627328    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6e9629-982e-4da4-a70d-434e94785168-plugins-config-volume\") pod \"5c6e9629-982e-4da4-a70d-434e94785168\" (UID: \"5c6e9629-982e-4da4-a70d-434e94785168\") "
Sep 15 19:10:23.627521 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.627378    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg8l2\" (UniqueName: \"kubernetes.io/projected/5c6e9629-982e-4da4-a70d-434e94785168-kube-api-access-zg8l2\") pod \"5c6e9629-982e-4da4-a70d-434e94785168\" (UID: \"5c6e9629-982e-4da4-a70d-434e94785168\") "
Sep 15 19:10:23.627866 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.627836    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6e9629-982e-4da4-a70d-434e94785168-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c6e9629-982e-4da4-a70d-434e94785168" (UID: "5c6e9629-982e-4da4-a70d-434e94785168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:10:23.629521 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.629483    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6e9629-982e-4da4-a70d-434e94785168-kube-api-access-zg8l2" (OuterVolumeSpecName: "kube-api-access-zg8l2") pod "5c6e9629-982e-4da4-a70d-434e94785168" (UID: "5c6e9629-982e-4da4-a70d-434e94785168"). InnerVolumeSpecName "kube-api-access-zg8l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:10:23.728658 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.728623    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6e9629-982e-4da4-a70d-434e94785168-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:10:23.728797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:23.728673    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg8l2\" (UniqueName: \"kubernetes.io/projected/5c6e9629-982e-4da4-a70d-434e94785168-kube-api-access-zg8l2\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:10:24.113187 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.113152    2556 generic.go:356] "Generic (PLEG): container finished" podID="5c6e9629-982e-4da4-a70d-434e94785168" containerID="b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0" exitCode=0
Sep 15 19:10:24.113390 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.113205    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"
Sep 15 19:10:24.113390 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.113216    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9" event={"ID":"5c6e9629-982e-4da4-a70d-434e94785168","Type":"ContainerDied","Data":"b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0"}
Sep 15 19:10:24.113390 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.113279    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9" event={"ID":"5c6e9629-982e-4da4-a70d-434e94785168","Type":"ContainerDied","Data":"b9af6da6a8554464c2173d0e4aad79327b4fd35ab60feed18c6ce2e43a605875"}
Sep 15 19:10:24.113390 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.113300    2556 scope.go:117] "RemoveContainer" containerID="b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0"
Sep 15 19:10:24.122684 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.122662    2556 scope.go:117] "RemoveContainer" containerID="b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0"
Sep 15 19:10:24.122983 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:10:24.122964    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0\": container with ID starting with b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0 not found: ID does not exist" containerID="b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0"
Sep 15 19:10:24.123052 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.122991    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0"} err="failed to get container status \"b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0\": rpc error: code = NotFound desc = could not find container \"b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0\": container with ID starting with b6637a1ee7280f3aeb10634a62233ba06340ced405424cc9738ec7d3cfc6a0e0 not found: ID does not exist"
Sep 15 19:10:24.139207 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.139184    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"]
Sep 15 19:10:24.145582 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:24.145558    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-282a6587-bf4d57449-wfcv9"]
Sep 15 19:10:25.679239 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:25.679209    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c6e9629-982e-4da4-a70d-434e94785168" path="/var/lib/kubelet/pods/5c6e9629-982e-4da4-a70d-434e94785168/volumes"
Sep 15 19:10:41.476234 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.476165    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"]
Sep 15 19:10:41.476828 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.476794    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6e9629-982e-4da4-a70d-434e94785168" containerName="payload-processing"
Sep 15 19:10:41.476828 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.476817    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6e9629-982e-4da4-a70d-434e94785168" containerName="payload-processing"
Sep 15 19:10:41.476977 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.476914    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6e9629-982e-4da4-a70d-434e94785168" containerName="payload-processing"
Sep 15 19:10:41.480049 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.480029    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.482470 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.482451    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"premium-w3-f0ad35-kserve-self-signed-certs\""
Sep 15 19:10:41.490015 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.489986    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"]
Sep 15 19:10:41.592768 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.592734    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-tmp-dir\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.592937 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.592847    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9r5g\" (UniqueName: \"kubernetes.io/projected/55635747-71db-4cb1-b0b2-c6b07f683278-kube-api-access-t9r5g\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.592937 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.592874    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-home\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.592937 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.592895    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55635747-71db-4cb1-b0b2-c6b07f683278-tls-certs\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.592937 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.592912    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-dshm\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.593131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.592944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-model-cache\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694102 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694067    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-model-cache\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694267 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694196    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-tmp-dir\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694331 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694287    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9r5g\" (UniqueName: \"kubernetes.io/projected/55635747-71db-4cb1-b0b2-c6b07f683278-kube-api-access-t9r5g\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694331 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694314    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-home\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694423 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55635747-71db-4cb1-b0b2-c6b07f683278-tls-certs\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694423 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694368    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-dshm\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694524 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694455    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-model-cache\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694524 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694507    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-tmp-dir\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.694657 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.694615    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-home\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.696546 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.696524    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-dshm\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.696680 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.696656    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"premium-w3-f0ad35-kserve-self-signed-certs\""
Sep 15 19:10:41.702367 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.702341    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9r5g\" (UniqueName: \"kubernetes.io/projected/55635747-71db-4cb1-b0b2-c6b07f683278-kube-api-access-t9r5g\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.707126 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.707104    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55635747-71db-4cb1-b0b2-c6b07f683278-tls-certs\") pod \"premium-w3-f0ad35-kserve-748b7d4848-kh6zl\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") " pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:41.755869 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.755694    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:10:41.766223 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.758070    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:10:41.766223 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.760877    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:10:41.766223 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.763486    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:10:41.791580 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:41.791553    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:42.062604 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:42.062580    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"]
Sep 15 19:10:42.064190 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:10:42.064157    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55635747_71db_4cb1_b0b2_c6b07f683278.slice/crio-82b02d4911a54c04f606ce6ee52f2cef0e50c56384749b62707f6bdeed2cac31 WatchSource:0}: Error finding container 82b02d4911a54c04f606ce6ee52f2cef0e50c56384749b62707f6bdeed2cac31: Status 404 returned error can't find the container with id 82b02d4911a54c04f606ce6ee52f2cef0e50c56384749b62707f6bdeed2cac31
Sep 15 19:10:42.209511 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:42.209476    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl" event={"ID":"55635747-71db-4cb1-b0b2-c6b07f683278","Type":"ContainerStarted","Data":"f7eebb881906a7ccdc3d8db454a4331e0df8dd0c6bb121298c9aa7346f44b9b9"}
Sep 15 19:10:42.209699 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:42.209520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl" event={"ID":"55635747-71db-4cb1-b0b2-c6b07f683278","Type":"ContainerStarted","Data":"82b02d4911a54c04f606ce6ee52f2cef0e50c56384749b62707f6bdeed2cac31"}
Sep 15 19:10:42.228653 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:42.228597    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl" podStartSLOduration=1.228579298 podStartE2EDuration="1.228579298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:42.225845085 +0000 UTC m=+1201.081698352" watchObservedRunningTime="2026-09-15 19:10:42.228579298 +0000 UTC m=+1201.084432566"
Sep 15 19:10:42.672296 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:42.672264    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:10:51.792171 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:51.792131    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:51.792598 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:51.792189    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:51.805377 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:51.805355    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:52.276227 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:52.276198    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:10:55.672550 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:10:55.672518    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:11:39.477240 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.477160    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"]
Sep 15 19:11:39.480795 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.480769    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.484928 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.484897    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"distinct-w3-f0ad35-kserve-self-signed-certs\""
Sep 15 19:11:39.493452 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.493423    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"]
Sep 15 19:11:39.660788 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.660750    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-home\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.660960 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.660806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-tmp-dir\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.660960 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.660902    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f004b09b-d718-44d2-a4a9-91977925ed3b-tls-certs\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.660960 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.660936    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdw7h\" (UniqueName: \"kubernetes.io/projected/f004b09b-d718-44d2-a4a9-91977925ed3b-kube-api-access-pdw7h\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.661133 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.661058    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-dshm\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.661133 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.661104    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-model-cache\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.761796 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.761719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-home\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.761796 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.761774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-tmp-dir\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.761978 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.761812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f004b09b-d718-44d2-a4a9-91977925ed3b-tls-certs\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.761978 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.761846    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdw7h\" (UniqueName: \"kubernetes.io/projected/f004b09b-d718-44d2-a4a9-91977925ed3b-kube-api-access-pdw7h\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.761978 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.761935    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-dshm\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.761978 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.761968    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-model-cache\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.762238 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.762188    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-home\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.762306 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.762228    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-tmp-dir\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.762363 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.762349    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-model-cache\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.764356 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.764328    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-dshm\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.764553 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.764533    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f004b09b-d718-44d2-a4a9-91977925ed3b-tls-certs\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.769092 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.769073    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdw7h\" (UniqueName: \"kubernetes.io/projected/f004b09b-d718-44d2-a4a9-91977925ed3b-kube-api-access-pdw7h\") pod \"distinct-w3-f0ad35-kserve-7555d5778c-5qfdd\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:39.796532 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:39.796509    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:40.068355 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:40.068331    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"]
Sep 15 19:11:40.070741 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:11:40.070706    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf004b09b_d718_44d2_a4a9_91977925ed3b.slice/crio-07af0f165bcf6ac4156ddded2a5205fcca9f39c5c685976e7168d95420fa4de9 WatchSource:0}: Error finding container 07af0f165bcf6ac4156ddded2a5205fcca9f39c5c685976e7168d95420fa4de9: Status 404 returned error can't find the container with id 07af0f165bcf6ac4156ddded2a5205fcca9f39c5c685976e7168d95420fa4de9
Sep 15 19:11:40.522350 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:40.522312    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd" event={"ID":"f004b09b-d718-44d2-a4a9-91977925ed3b","Type":"ContainerStarted","Data":"01be4caf2fe743a46432f8a882e4796143c60f283f1ce523e519b0183cf0e73e"}
Sep 15 19:11:40.522760 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:40.522361    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd" event={"ID":"f004b09b-d718-44d2-a4a9-91977925ed3b","Type":"ContainerStarted","Data":"07af0f165bcf6ac4156ddded2a5205fcca9f39c5c685976e7168d95420fa4de9"}
Sep 15 19:11:40.540030 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:40.539982    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd" podStartSLOduration=1.539966237 podStartE2EDuration="1.539966237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:40.538211804 +0000 UTC m=+1259.394065071" watchObservedRunningTime="2026-09-15 19:11:40.539966237 +0000 UTC m=+1259.395819546"
Sep 15 19:11:49.796948 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:49.796907    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:49.796948 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:49.796954    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:49.809730 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:49.809703    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:11:50.588121 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:11:50.588081    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:12:07.672224 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:12:07.672195    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:12:09.672637 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:12:09.672603    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:13:18.671729 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:18.671631    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:13:24.672523 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:24.672487    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:13:39.125316 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.125281    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"]
Sep 15 19:13:39.129515 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.129470    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.132034 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.132010    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-1b0e3f-serving-cert\""
Sep 15 19:13:39.137749 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.137712    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"]
Sep 15 19:13:39.242594 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.242561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkr5\" (UniqueName: \"kubernetes.io/projected/6df2bd20-a2e1-4ca2-9d02-0934804bab25-kube-api-access-hhkr5\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.242781 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.242627    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.242781 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.242694    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.242781 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.242766    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df2bd20-a2e1-4ca2-9d02-0934804bab25-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.344187 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.344143    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.344392 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.344218    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.344392 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.344252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df2bd20-a2e1-4ca2-9d02-0934804bab25-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.344392 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:13:39.344308    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-serving-cert: secret "maas-api-e2e-shared-b-w1-1b0e3f-serving-cert" not found
Sep 15 19:13:39.344392 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.344330    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkr5\" (UniqueName: \"kubernetes.io/projected/6df2bd20-a2e1-4ca2-9d02-0934804bab25-kube-api-access-hhkr5\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.344392 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:13:39.344373    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls podName:6df2bd20-a2e1-4ca2-9d02-0934804bab25 nodeName:}" failed. No retries permitted until 2026-09-15 19:13:39.844353066 +0000 UTC m=+1378.700206317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls") pod "maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4" (UID: "6df2bd20-a2e1-4ca2-9d02-0934804bab25") : secret "maas-api-e2e-shared-b-w1-1b0e3f-serving-cert" not found
Sep 15 19:13:39.345038 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.345016    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df2bd20-a2e1-4ca2-9d02-0934804bab25-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.346722 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.346692    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.352668 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.352632    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkr5\" (UniqueName: \"kubernetes.io/projected/6df2bd20-a2e1-4ca2-9d02-0934804bab25-kube-api-access-hhkr5\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.824269 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.824240    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"]
Sep 15 19:13:39.827997 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.827981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:39.830168 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.830145    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-1b0e3f\""
Sep 15 19:13:39.830469 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.830452    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-1b0e3f-dockercfg-hj6h2\""
Sep 15 19:13:39.835706 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.835681    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"]
Sep 15 19:13:39.850957 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.850928    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.853420 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.853395    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:39.951878 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.951843    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aab0344-ae04-4211-b2cf-6366e000d19d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g\" (UID: \"6aab0344-ae04-4211-b2cf-6366e000d19d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:39.952031 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:39.951907    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrng9\" (UniqueName: \"kubernetes.io/projected/6aab0344-ae04-4211-b2cf-6366e000d19d-kube-api-access-lrng9\") pod \"payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g\" (UID: \"6aab0344-ae04-4211-b2cf-6366e000d19d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:40.042821 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.042787    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:40.052731 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.052703    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aab0344-ae04-4211-b2cf-6366e000d19d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g\" (UID: \"6aab0344-ae04-4211-b2cf-6366e000d19d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:40.052862 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.052768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrng9\" (UniqueName: \"kubernetes.io/projected/6aab0344-ae04-4211-b2cf-6366e000d19d-kube-api-access-lrng9\") pod \"payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g\" (UID: \"6aab0344-ae04-4211-b2cf-6366e000d19d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:40.053404 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.053386    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aab0344-ae04-4211-b2cf-6366e000d19d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g\" (UID: \"6aab0344-ae04-4211-b2cf-6366e000d19d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:40.060774 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.060750    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrng9\" (UniqueName: \"kubernetes.io/projected/6aab0344-ae04-4211-b2cf-6366e000d19d-kube-api-access-lrng9\") pod \"payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g\" (UID: \"6aab0344-ae04-4211-b2cf-6366e000d19d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:40.140817 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.140784    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:40.317616 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.317586    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"]
Sep 15 19:13:40.318653 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:13:40.318615    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df2bd20_a2e1_4ca2_9d02_0934804bab25.slice/crio-69f04724cf60f3d095a0c26b31478cdc66495d6533b063fd6f240c3473918a2c WatchSource:0}: Error finding container 69f04724cf60f3d095a0c26b31478cdc66495d6533b063fd6f240c3473918a2c: Status 404 returned error can't find the container with id 69f04724cf60f3d095a0c26b31478cdc66495d6533b063fd6f240c3473918a2c
Sep 15 19:13:40.320024 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.320007    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:13:40.420684 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:40.420655    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"]
Sep 15 19:13:40.422463 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:13:40.422440    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aab0344_ae04_4211_b2cf_6366e000d19d.slice/crio-5a6c4bf8f5fc2bb3841392fe4860c49758ad932840587d9c2f2b4735e8b604b6 WatchSource:0}: Error finding container 5a6c4bf8f5fc2bb3841392fe4860c49758ad932840587d9c2f2b4735e8b604b6: Status 404 returned error can't find the container with id 5a6c4bf8f5fc2bb3841392fe4860c49758ad932840587d9c2f2b4735e8b604b6
Sep 15 19:13:41.176731 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:41.176690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4" event={"ID":"6df2bd20-a2e1-4ca2-9d02-0934804bab25","Type":"ContainerStarted","Data":"2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb"}
Sep 15 19:13:41.177162 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:41.176758    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4" event={"ID":"6df2bd20-a2e1-4ca2-9d02-0934804bab25","Type":"ContainerStarted","Data":"69f04724cf60f3d095a0c26b31478cdc66495d6533b063fd6f240c3473918a2c"}
Sep 15 19:13:41.177162 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:41.176816    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:41.178387 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:41.178365    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g" event={"ID":"6aab0344-ae04-4211-b2cf-6366e000d19d","Type":"ContainerStarted","Data":"cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430"}
Sep 15 19:13:41.178488 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:41.178403    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g" event={"ID":"6aab0344-ae04-4211-b2cf-6366e000d19d","Type":"ContainerStarted","Data":"5a6c4bf8f5fc2bb3841392fe4860c49758ad932840587d9c2f2b4735e8b604b6"}
Sep 15 19:13:41.178488 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:41.178475    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:13:41.193503 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:41.193429    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4" podStartSLOduration=1.8547714389999999 podStartE2EDuration="2.193416634s" totalImagesPullingTime="338.645195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:41.190678671 +0000 UTC m=+1380.046531939" watchObservedRunningTime="2026-09-15 19:13:41.193416634 +0000 UTC m=+1380.049269903"
Sep 15 19:13:41.207059 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:41.207016    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g" podStartSLOduration=2.207001591 podStartE2EDuration="2.207001591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:41.205354404 +0000 UTC m=+1380.061207673" watchObservedRunningTime="2026-09-15 19:13:41.207001591 +0000 UTC m=+1380.062854908"
Sep 15 19:13:47.189463 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:47.189433    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:13:52.185981 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:13:52.185952    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:14:09.356826 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.356792    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8999-400ced"]
Sep 15 19:14:09.360738 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.360722    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-400ced"
Sep 15 19:14:09.363111 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.363076    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgkcn\""
Sep 15 19:14:09.365954 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.365935    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8999-400ced"]
Sep 15 19:14:09.468270 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.468243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82fm\" (UniqueName: \"kubernetes.io/projected/81c264f8-ca39-4921-9f2b-b79e2be4a768-kube-api-access-w82fm\") pod \"test-curl-8999-400ced\" (UID: \"81c264f8-ca39-4921-9f2b-b79e2be4a768\") " pod="opendatahub/test-curl-8999-400ced"
Sep 15 19:14:09.569086 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.569056    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82fm\" (UniqueName: \"kubernetes.io/projected/81c264f8-ca39-4921-9f2b-b79e2be4a768-kube-api-access-w82fm\") pod \"test-curl-8999-400ced\" (UID: \"81c264f8-ca39-4921-9f2b-b79e2be4a768\") " pod="opendatahub/test-curl-8999-400ced"
Sep 15 19:14:09.577281 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.577256    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82fm\" (UniqueName: \"kubernetes.io/projected/81c264f8-ca39-4921-9f2b-b79e2be4a768-kube-api-access-w82fm\") pod \"test-curl-8999-400ced\" (UID: \"81c264f8-ca39-4921-9f2b-b79e2be4a768\") " pod="opendatahub/test-curl-8999-400ced"
Sep 15 19:14:09.671657 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.671617    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-400ced"
Sep 15 19:14:09.944442 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:09.944418    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8999-400ced"]
Sep 15 19:14:09.946590 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:14:09.946563    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81c264f8_ca39_4921_9f2b_b79e2be4a768.slice/crio-07227e046294efbbf72ba51d0cbf2dad55160c63ba9faada193b2f1266613d99 WatchSource:0}: Error finding container 07227e046294efbbf72ba51d0cbf2dad55160c63ba9faada193b2f1266613d99: Status 404 returned error can't find the container with id 07227e046294efbbf72ba51d0cbf2dad55160c63ba9faada193b2f1266613d99
Sep 15 19:14:10.339800 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:10.339715    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8999-400ced" event={"ID":"81c264f8-ca39-4921-9f2b-b79e2be4a768","Type":"ContainerStarted","Data":"a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1"}
Sep 15 19:14:10.339800 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:10.339755    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8999-400ced" event={"ID":"81c264f8-ca39-4921-9f2b-b79e2be4a768","Type":"ContainerStarted","Data":"07227e046294efbbf72ba51d0cbf2dad55160c63ba9faada193b2f1266613d99"}
Sep 15 19:14:10.362254 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:10.362197    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8999-400ced" podStartSLOduration=1.362183833 podStartE2EDuration="1.362183833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:10.359358898 +0000 UTC m=+1409.215212165" watchObservedRunningTime="2026-09-15 19:14:10.362183833 +0000 UTC m=+1409.218037101"
Sep 15 19:14:10.938011 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:10.937924    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8999-400ced"]
Sep 15 19:14:10.941239 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:10.941208    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8999-400ced"]
Sep 15 19:14:11.345577 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:14:11.345549    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8999-400ced" container="test-curl-8999-400ced"
Sep 15 19:14:12.350828 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:12.350793    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8999-400ced" podUID="81c264f8-ca39-4921-9f2b-b79e2be4a768" containerName="test-curl-8999-400ced" containerID="cri-o://a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1" gracePeriod=2
Sep 15 19:14:14.541044 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:14.541014    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-400ced"
Sep 15 19:14:14.622811 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:14.622781    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w82fm\" (UniqueName: \"kubernetes.io/projected/81c264f8-ca39-4921-9f2b-b79e2be4a768-kube-api-access-w82fm\") pod \"81c264f8-ca39-4921-9f2b-b79e2be4a768\" (UID: \"81c264f8-ca39-4921-9f2b-b79e2be4a768\") "
Sep 15 19:14:14.624920 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:14.624885    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c264f8-ca39-4921-9f2b-b79e2be4a768-kube-api-access-w82fm" (OuterVolumeSpecName: "kube-api-access-w82fm") pod "81c264f8-ca39-4921-9f2b-b79e2be4a768" (UID: "81c264f8-ca39-4921-9f2b-b79e2be4a768"). InnerVolumeSpecName "kube-api-access-w82fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:14.723909 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:14.723869    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w82fm\" (UniqueName: \"kubernetes.io/projected/81c264f8-ca39-4921-9f2b-b79e2be4a768-kube-api-access-w82fm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:14:15.367749 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:15.367712    2556 generic.go:356] "Generic (PLEG): container finished" podID="81c264f8-ca39-4921-9f2b-b79e2be4a768" containerID="a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1" exitCode=137
Sep 15 19:14:15.367924 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:15.367758    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-400ced"
Sep 15 19:14:15.367924 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:15.367774    2556 scope.go:117] "RemoveContainer" containerID="a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1"
Sep 15 19:14:15.377764 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:15.377744    2556 scope.go:117] "RemoveContainer" containerID="a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1"
Sep 15 19:14:15.378089 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:14:15.378070    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1\": container with ID starting with a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1 not found: ID does not exist" containerID="a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1"
Sep 15 19:14:15.378165 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:15.378102    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1"} err="failed to get container status \"a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1\": rpc error: code = NotFound desc = could not find container \"a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1\": container with ID starting with a297bf054be82e902e374caaf40e91c9853bcc3ccdc555bea24d053143d557f1 not found: ID does not exist"
Sep 15 19:14:15.378628 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:14:15.378604    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8999-400ced\" is forbidden: User \"system:node:ip-10-0-140-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-183.ec2.internal' and this object" podUID="81c264f8-ca39-4921-9f2b-b79e2be4a768" pod="opendatahub/test-curl-8999-400ced"
Sep 15 19:14:15.682198 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:15.682154    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c264f8-ca39-4921-9f2b-b79e2be4a768" path="/var/lib/kubelet/pods/81c264f8-ca39-4921-9f2b-b79e2be4a768/volumes"
Sep 15 19:14:20.621520 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.621487    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"]
Sep 15 19:14:20.621980 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.621960    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c264f8-ca39-4921-9f2b-b79e2be4a768" containerName="test-curl-8999-400ced"
Sep 15 19:14:20.621980 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.621981    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c264f8-ca39-4921-9f2b-b79e2be4a768" containerName="test-curl-8999-400ced"
Sep 15 19:14:20.622074 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.622065    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c264f8-ca39-4921-9f2b-b79e2be4a768" containerName="test-curl-8999-400ced"
Sep 15 19:14:20.634352 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.634325    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:20.636722 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.636696    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-17c562de\""
Sep 15 19:14:20.636894 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.636870    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-17c562de-dockercfg-wzjcr\""
Sep 15 19:14:20.638730 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.638698    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"]
Sep 15 19:14:20.785933 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.785900    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a9900-3226-4df5-b57c-62b5c2919583-plugins-config-volume\") pod \"payload-processing-e2e-mt-17c562de-677f698857-ckph8\" (UID: \"c15a9900-3226-4df5-b57c-62b5c2919583\") " pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:20.786114 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.786069    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2hc\" (UniqueName: \"kubernetes.io/projected/c15a9900-3226-4df5-b57c-62b5c2919583-kube-api-access-rz2hc\") pod \"payload-processing-e2e-mt-17c562de-677f698857-ckph8\" (UID: \"c15a9900-3226-4df5-b57c-62b5c2919583\") " pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:20.886877 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.886794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a9900-3226-4df5-b57c-62b5c2919583-plugins-config-volume\") pod \"payload-processing-e2e-mt-17c562de-677f698857-ckph8\" (UID: \"c15a9900-3226-4df5-b57c-62b5c2919583\") " pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:20.887023 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.886894    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2hc\" (UniqueName: \"kubernetes.io/projected/c15a9900-3226-4df5-b57c-62b5c2919583-kube-api-access-rz2hc\") pod \"payload-processing-e2e-mt-17c562de-677f698857-ckph8\" (UID: \"c15a9900-3226-4df5-b57c-62b5c2919583\") " pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:20.887535 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.887516    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a9900-3226-4df5-b57c-62b5c2919583-plugins-config-volume\") pod \"payload-processing-e2e-mt-17c562de-677f698857-ckph8\" (UID: \"c15a9900-3226-4df5-b57c-62b5c2919583\") " pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:20.894260 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.894230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2hc\" (UniqueName: \"kubernetes.io/projected/c15a9900-3226-4df5-b57c-62b5c2919583-kube-api-access-rz2hc\") pod \"payload-processing-e2e-mt-17c562de-677f698857-ckph8\" (UID: \"c15a9900-3226-4df5-b57c-62b5c2919583\") " pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:20.947061 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:20.947030    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:21.219432 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:21.219408    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"]
Sep 15 19:14:21.221404 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:14:21.221382    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15a9900_3226_4df5_b57c_62b5c2919583.slice/crio-30fb3273a16485113f35472604ffaec1851c7582e01ef57a86c070f6716013a6 WatchSource:0}: Error finding container 30fb3273a16485113f35472604ffaec1851c7582e01ef57a86c070f6716013a6: Status 404 returned error can't find the container with id 30fb3273a16485113f35472604ffaec1851c7582e01ef57a86c070f6716013a6
Sep 15 19:14:21.403926 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:21.403892    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8" event={"ID":"c15a9900-3226-4df5-b57c-62b5c2919583","Type":"ContainerStarted","Data":"f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba"}
Sep 15 19:14:21.404119 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:21.403943    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8" event={"ID":"c15a9900-3226-4df5-b57c-62b5c2919583","Type":"ContainerStarted","Data":"30fb3273a16485113f35472604ffaec1851c7582e01ef57a86c070f6716013a6"}
Sep 15 19:14:21.404119 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:21.404008    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:21.430696 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:21.430636    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8" podStartSLOduration=1.430620404 podStartE2EDuration="1.430620404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:21.427995235 +0000 UTC m=+1420.283848504" watchObservedRunningTime="2026-09-15 19:14:21.430620404 +0000 UTC m=+1420.286473690"
Sep 15 19:14:21.675519 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:21.675486    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:14:32.412167 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:32.412135    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:14:32.672848 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:32.672761    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:14:34.735221 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:34.735128    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"]
Sep 15 19:14:34.735755 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:34.735344    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf" podUID="d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" containerName="maas-api" containerID="cri-o://8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33" gracePeriod=30
Sep 15 19:14:34.798374 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:34.798334    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"]
Sep 15 19:14:34.798619 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:34.798589    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5" podUID="f8847446-857a-4741-a03f-076891defe53" containerName="payload-pre-processing" containerID="cri-o://2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea" gracePeriod=30
Sep 15 19:14:35.057410 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.057381    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:14:35.086753 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.086728    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:14:35.129935 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.129903    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-metrics-certs\") pod \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") "
Sep 15 19:14:35.130106 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.129956    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmm7h\" (UniqueName: \"kubernetes.io/projected/f8847446-857a-4741-a03f-076891defe53-kube-api-access-jmm7h\") pod \"f8847446-857a-4741-a03f-076891defe53\" (UID: \"f8847446-857a-4741-a03f-076891defe53\") "
Sep 15 19:14:35.130106 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.129985    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-maas-api-tls\") pod \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") "
Sep 15 19:14:35.130106 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.130029    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-openshift-service-ca\") pod \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") "
Sep 15 19:14:35.130106 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.130101    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8847446-857a-4741-a03f-076891defe53-plugins-config-volume\") pod \"f8847446-857a-4741-a03f-076891defe53\" (UID: \"f8847446-857a-4741-a03f-076891defe53\") "
Sep 15 19:14:35.130328 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.130133    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7679\" (UniqueName: \"kubernetes.io/projected/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-kube-api-access-z7679\") pod \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\" (UID: \"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa\") "
Sep 15 19:14:35.130487 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.130459    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" (UID: "d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:14:35.130576 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.130498    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8847446-857a-4741-a03f-076891defe53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8847446-857a-4741-a03f-076891defe53" (UID: "f8847446-857a-4741-a03f-076891defe53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:14:35.132181 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.132157    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" (UID: "d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:35.132262 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.132198    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" (UID: "d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:35.132262 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.132212    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8847446-857a-4741-a03f-076891defe53-kube-api-access-jmm7h" (OuterVolumeSpecName: "kube-api-access-jmm7h") pod "f8847446-857a-4741-a03f-076891defe53" (UID: "f8847446-857a-4741-a03f-076891defe53"). InnerVolumeSpecName "kube-api-access-jmm7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:35.132262 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.132230    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-kube-api-access-z7679" (OuterVolumeSpecName: "kube-api-access-z7679") pod "d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" (UID: "d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa"). InnerVolumeSpecName "kube-api-access-z7679". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:35.231062 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.231028    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7679\" (UniqueName: \"kubernetes.io/projected/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-kube-api-access-z7679\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:14:35.231062 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.231058    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:14:35.231062 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.231067    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmm7h\" (UniqueName: \"kubernetes.io/projected/f8847446-857a-4741-a03f-076891defe53-kube-api-access-jmm7h\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:14:35.231285 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.231076    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-maas-api-tls\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:14:35.231285 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.231085    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa-openshift-service-ca\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:14:35.231285 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.231100    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8847446-857a-4741-a03f-076891defe53-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:14:35.489256 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.489219    2556 generic.go:356] "Generic (PLEG): container finished" podID="d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" containerID="8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33" exitCode=0
Sep 15 19:14:35.489449 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.489274    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"
Sep 15 19:14:35.489449 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.489303    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf" event={"ID":"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa","Type":"ContainerDied","Data":"8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33"}
Sep 15 19:14:35.489449 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.489398    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf" event={"ID":"d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa","Type":"ContainerDied","Data":"1f8418a323e7189bce9f9e1a11e2676d7c8a7db2fe90b7b16dc4f8c7939e9e4d"}
Sep 15 19:14:35.489449 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.489427    2556 scope.go:117] "RemoveContainer" containerID="8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33"
Sep 15 19:14:35.490757 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.490733    2556 generic.go:356] "Generic (PLEG): container finished" podID="f8847446-857a-4741-a03f-076891defe53" containerID="2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea" exitCode=0
Sep 15 19:14:35.490887 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.490763    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5" event={"ID":"f8847446-857a-4741-a03f-076891defe53","Type":"ContainerDied","Data":"2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea"}
Sep 15 19:14:35.490887 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.490787    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"
Sep 15 19:14:35.490887 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.490812    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5" event={"ID":"f8847446-857a-4741-a03f-076891defe53","Type":"ContainerDied","Data":"7e572d69bdba5d3d4fd329c9da65b0979b86b52b6c230a2ca015c1ae29636362"}
Sep 15 19:14:35.500907 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.500891    2556 scope.go:117] "RemoveContainer" containerID="8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33"
Sep 15 19:14:35.501198 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:14:35.501168    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33\": container with ID starting with 8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33 not found: ID does not exist" containerID="8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33"
Sep 15 19:14:35.501284 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.501210    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33"} err="failed to get container status \"8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33\": rpc error: code = NotFound desc = could not find container \"8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33\": container with ID starting with 8a4fb99562cb14bc3a2cf18ec05385df4aa07702195a222bee6977e0bec99e33 not found: ID does not exist"
Sep 15 19:14:35.501284 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.501232    2556 scope.go:117] "RemoveContainer" containerID="2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea"
Sep 15 19:14:35.509268 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.509254    2556 scope.go:117] "RemoveContainer" containerID="2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea"
Sep 15 19:14:35.509535 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:14:35.509516    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea\": container with ID starting with 2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea not found: ID does not exist" containerID="2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea"
Sep 15 19:14:35.509598 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.509544    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea"} err="failed to get container status \"2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea\": rpc error: code = NotFound desc = could not find container \"2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea\": container with ID starting with 2b9eb6d65bd893fca45ca1105060db53c5656c5bef8783971299b1ce461567ea not found: ID does not exist"
Sep 15 19:14:35.513102 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.513082    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"]
Sep 15 19:14:35.518571 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.518546    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9de44d-7c66788bf7-4g9pf"]
Sep 15 19:14:35.527993 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.527973    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"]
Sep 15 19:14:35.531772 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.531752    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9de44d-66d4dbb58-wxsd5"]
Sep 15 19:14:35.679026 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.678991    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" path="/var/lib/kubelet/pods/d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa/volumes"
Sep 15 19:14:35.679378 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:14:35.679365    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8847446-857a-4741-a03f-076891defe53" path="/var/lib/kubelet/pods/f8847446-857a-4741-a03f-076891defe53/volumes"
Sep 15 19:15:00.182369 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.182325    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"]
Sep 15 19:15:00.183018 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.182951    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8847446-857a-4741-a03f-076891defe53" containerName="payload-pre-processing"
Sep 15 19:15:00.183018 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.182967    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8847446-857a-4741-a03f-076891defe53" containerName="payload-pre-processing"
Sep 15 19:15:00.183018 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.182989    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" containerName="maas-api"
Sep 15 19:15:00.183018 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.182994    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" containerName="maas-api"
Sep 15 19:15:00.183257 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.183087    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8847446-857a-4741-a03f-076891defe53" containerName="payload-pre-processing"
Sep 15 19:15:00.183257 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.183098    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f4e1c0-3ca9-4c70-8283-1f83791d0bfa" containerName="maas-api"
Sep 15 19:15:00.187886 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.187865    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"
Sep 15 19:15:00.190498 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.190472    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:15:00.196075 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.196049    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"]
Sep 15 19:15:00.200375 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.200355    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"
Sep 15 19:15:00.472434 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.472405    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"]
Sep 15 19:15:00.474005 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:15:00.473981    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12fc6c6_51e9_4d5e_bd77_40429e1edee4.slice/crio-a328350ebc9f36097352bf52a9f28c2da04d6725ddaa1306d9abd0f07a3bd5d2 WatchSource:0}: Error finding container a328350ebc9f36097352bf52a9f28c2da04d6725ddaa1306d9abd0f07a3bd5d2: Status 404 returned error can't find the container with id a328350ebc9f36097352bf52a9f28c2da04d6725ddaa1306d9abd0f07a3bd5d2
Sep 15 19:15:00.627450 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:00.627417    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22" event={"ID":"d12fc6c6-51e9-4d5e-bd77-40429e1edee4","Type":"ContainerStarted","Data":"a328350ebc9f36097352bf52a9f28c2da04d6725ddaa1306d9abd0f07a3bd5d2"}
Sep 15 19:15:03.649263 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:03.649227    2556 generic.go:356] "Generic (PLEG): container finished" podID="d12fc6c6-51e9-4d5e-bd77-40429e1edee4" containerID="3fa8b31a90e89b132f3dade49323426a936636ca46cc44b60924bc5a326754a9" exitCode=0
Sep 15 19:15:03.649602 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:03.649311    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22" event={"ID":"d12fc6c6-51e9-4d5e-bd77-40429e1edee4","Type":"ContainerDied","Data":"3fa8b31a90e89b132f3dade49323426a936636ca46cc44b60924bc5a326754a9"}
Sep 15 19:15:04.786780 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:04.786755    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"
Sep 15 19:15:05.662109 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:05.662077    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"
Sep 15 19:15:05.662279 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:05.662126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22" event={"ID":"d12fc6c6-51e9-4d5e-bd77-40429e1edee4","Type":"ContainerDied","Data":"a328350ebc9f36097352bf52a9f28c2da04d6725ddaa1306d9abd0f07a3bd5d2"}
Sep 15 19:15:05.662279 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:05.662174    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a328350ebc9f36097352bf52a9f28c2da04d6725ddaa1306d9abd0f07a3bd5d2"
Sep 15 19:15:16.740900 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.740862    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"]
Sep 15 19:15:16.741634 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.741608    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12fc6c6-51e9-4d5e-bd77-40429e1edee4" containerName="cleanup"
Sep 15 19:15:16.741843 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.741818    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12fc6c6-51e9-4d5e-bd77-40429e1edee4" containerName="cleanup"
Sep 15 19:15:16.742034 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.742019    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12fc6c6-51e9-4d5e-bd77-40429e1edee4" containerName="cleanup"
Sep 15 19:15:16.745455 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.745431    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.749976 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.749755    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"openshift-service-ca.crt\""
Sep 15 19:15:16.749976 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.749948    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"default-dockercfg-6plq7\""
Sep 15 19:15:16.750175 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.750016    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"kube-root-ca.crt\""
Sep 15 19:15:16.750175 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.750117    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"distinct-2-w0-900c88-kserve-self-signed-certs\""
Sep 15 19:15:16.765379 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.765354    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"]
Sep 15 19:15:16.842140 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.842107    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-model-cache\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.842273 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.842150    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-tmp-dir\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.842273 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.842171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bgs\" (UniqueName: \"kubernetes.io/projected/9ab597c1-ab83-411c-91b1-285d384a64da-kube-api-access-55bgs\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.842273 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.842235    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-home\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.842390 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.842287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab597c1-ab83-411c-91b1-285d384a64da-tls-certs\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.842390 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.842335    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-dshm\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.943342 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943311    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-home\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.943496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943356    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab597c1-ab83-411c-91b1-285d384a64da-tls-certs\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.943496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-dshm\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.943496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943426    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-model-cache\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.943496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943453    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-tmp-dir\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.943496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943469    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bgs\" (UniqueName: \"kubernetes.io/projected/9ab597c1-ab83-411c-91b1-285d384a64da-kube-api-access-55bgs\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.943818 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943777    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-home\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.943927 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-model-cache\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.944003 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.943929    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-tmp-dir\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.945760 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.945739    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-dshm\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.946029 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.946008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab597c1-ab83-411c-91b1-285d384a64da-tls-certs\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:16.951236 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:16.951217    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bgs\" (UniqueName: \"kubernetes.io/projected/9ab597c1-ab83-411c-91b1-285d384a64da-kube-api-access-55bgs\") pod \"distinct-2-w0-900c88-kserve-64995598cb-nxs65\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:17.056680 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:17.056582    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:17.334469 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:17.334444    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"]
Sep 15 19:15:17.335825 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:15:17.335799    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ab597c1_ab83_411c_91b1_285d384a64da.slice/crio-c51f2c21c20fec9b53b9e30ef5ef84f79ea949460e98a75e0649818fc137e747 WatchSource:0}: Error finding container c51f2c21c20fec9b53b9e30ef5ef84f79ea949460e98a75e0649818fc137e747: Status 404 returned error can't find the container with id c51f2c21c20fec9b53b9e30ef5ef84f79ea949460e98a75e0649818fc137e747
Sep 15 19:15:17.728817 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:17.728783    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65" event={"ID":"9ab597c1-ab83-411c-91b1-285d384a64da","Type":"ContainerStarted","Data":"d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca"}
Sep 15 19:15:17.728982 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:17.728825    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65" event={"ID":"9ab597c1-ab83-411c-91b1-285d384a64da","Type":"ContainerStarted","Data":"c51f2c21c20fec9b53b9e30ef5ef84f79ea949460e98a75e0649818fc137e747"}
Sep 15 19:15:17.747508 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:17.747458    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65" podStartSLOduration=1.747442682 podStartE2EDuration="1.747442682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:17.746018359 +0000 UTC m=+1476.601871628" watchObservedRunningTime="2026-09-15 19:15:17.747442682 +0000 UTC m=+1476.603295950"
Sep 15 19:15:27.056921 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:27.056880    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:27.057359 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:27.056938    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:27.069755 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:27.069732    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:27.806238 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:27.806209    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:15:28.616492 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.616462    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"]
Sep 15 19:15:28.616995 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.616747    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz" podUID="71be7022-f1b9-478c-8035-1d4ab85a0d7c" containerName="maas-api" containerID="cri-o://a2d6bf42e5f79222497ce722b65113912d665cf8138c8461f8c592623a020ac4" gracePeriod=30
Sep 15 19:15:28.803689 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.803626    2556 generic.go:356] "Generic (PLEG): container finished" podID="71be7022-f1b9-478c-8035-1d4ab85a0d7c" containerID="a2d6bf42e5f79222497ce722b65113912d665cf8138c8461f8c592623a020ac4" exitCode=0
Sep 15 19:15:28.803854 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.803794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz" event={"ID":"71be7022-f1b9-478c-8035-1d4ab85a0d7c","Type":"ContainerDied","Data":"a2d6bf42e5f79222497ce722b65113912d665cf8138c8461f8c592623a020ac4"}
Sep 15 19:15:28.874238 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.874209    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:15:28.964382 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.964349    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c42kh\" (UniqueName: \"kubernetes.io/projected/71be7022-f1b9-478c-8035-1d4ab85a0d7c-kube-api-access-c42kh\") pod \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") "
Sep 15 19:15:28.964561 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.964403    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls\") pod \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") "
Sep 15 19:15:28.964561 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.964440    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-metrics-certs\") pod \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") "
Sep 15 19:15:28.964561 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.964482    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be7022-f1b9-478c-8035-1d4ab85a0d7c-openshift-service-ca\") pod \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\" (UID: \"71be7022-f1b9-478c-8035-1d4ab85a0d7c\") "
Sep 15 19:15:28.964985 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.964955    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71be7022-f1b9-478c-8035-1d4ab85a0d7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71be7022-f1b9-478c-8035-1d4ab85a0d7c" (UID: "71be7022-f1b9-478c-8035-1d4ab85a0d7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:15:28.966615 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.966586    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71be7022-f1b9-478c-8035-1d4ab85a0d7c" (UID: "71be7022-f1b9-478c-8035-1d4ab85a0d7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:15:28.967021 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.966994    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71be7022-f1b9-478c-8035-1d4ab85a0d7c" (UID: "71be7022-f1b9-478c-8035-1d4ab85a0d7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:15:28.967021 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:28.967010    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71be7022-f1b9-478c-8035-1d4ab85a0d7c-kube-api-access-c42kh" (OuterVolumeSpecName: "kube-api-access-c42kh") pod "71be7022-f1b9-478c-8035-1d4ab85a0d7c" (UID: "71be7022-f1b9-478c-8035-1d4ab85a0d7c"). InnerVolumeSpecName "kube-api-access-c42kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:29.065911 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.065871    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c42kh\" (UniqueName: \"kubernetes.io/projected/71be7022-f1b9-478c-8035-1d4ab85a0d7c-kube-api-access-c42kh\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:15:29.065911 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.065910    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-maas-api-tls\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:15:29.066095 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.065926    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71be7022-f1b9-478c-8035-1d4ab85a0d7c-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:15:29.066095 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.065939    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be7022-f1b9-478c-8035-1d4ab85a0d7c-openshift-service-ca\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:15:29.811477 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.811444    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz" event={"ID":"71be7022-f1b9-478c-8035-1d4ab85a0d7c","Type":"ContainerDied","Data":"007d5c5b6fd681138de64efe8d086c26dbebb0e43b68ec35072574b9d17056e7"}
Sep 15 19:15:29.811477 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.811480    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"
Sep 15 19:15:29.811999 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.811505    2556 scope.go:117] "RemoveContainer" containerID="a2d6bf42e5f79222497ce722b65113912d665cf8138c8461f8c592623a020ac4"
Sep 15 19:15:29.829959 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.829935    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"]
Sep 15 19:15:29.833120 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:29.833099    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daa58-6c9db8cbcb-j6lxz"]
Sep 15 19:15:31.679005 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:31.678973    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71be7022-f1b9-478c-8035-1d4ab85a0d7c" path="/var/lib/kubelet/pods/71be7022-f1b9-478c-8035-1d4ab85a0d7c/volumes"
Sep 15 19:15:41.794377 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:41.794256    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:15:41.812019 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:41.799212    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:15:41.812019 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:41.799915    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:15:41.812019 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:41.805566    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:15:42.672057 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:42.672023    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:15:43.900195 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:43.900157    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"]
Sep 15 19:15:43.900652 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:43.900625    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71be7022-f1b9-478c-8035-1d4ab85a0d7c" containerName="maas-api"
Sep 15 19:15:43.900652 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:43.900652    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="71be7022-f1b9-478c-8035-1d4ab85a0d7c" containerName="maas-api"
Sep 15 19:15:43.900742 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:43.900732    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="71be7022-f1b9-478c-8035-1d4ab85a0d7c" containerName="maas-api"
Sep 15 19:15:43.903879 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:43.903863    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:43.906388 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:43.906366    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c6d10a\""
Sep 15 19:15:43.906507 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:43.906410    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c6d10a-dockercfg-gjh5f\""
Sep 15 19:15:43.914579 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:43.914558    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"]
Sep 15 19:15:44.021994 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.021964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10ff7a4-0dbb-4959-bf99-ef542190ec08-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k\" (UID: \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:44.022152 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.022098    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjtm\" (UniqueName: \"kubernetes.io/projected/e10ff7a4-0dbb-4959-bf99-ef542190ec08-kube-api-access-ptjtm\") pod \"payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k\" (UID: \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:44.123598 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.123562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10ff7a4-0dbb-4959-bf99-ef542190ec08-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k\" (UID: \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:44.123812 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.123792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjtm\" (UniqueName: \"kubernetes.io/projected/e10ff7a4-0dbb-4959-bf99-ef542190ec08-kube-api-access-ptjtm\") pod \"payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k\" (UID: \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:44.124505 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.124484    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10ff7a4-0dbb-4959-bf99-ef542190ec08-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k\" (UID: \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:44.132169 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.132142    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjtm\" (UniqueName: \"kubernetes.io/projected/e10ff7a4-0dbb-4959-bf99-ef542190ec08-kube-api-access-ptjtm\") pod \"payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k\" (UID: \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:44.216941 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.216865    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:44.493594 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.493562    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"]
Sep 15 19:15:44.495622 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:15:44.495591    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10ff7a4_0dbb_4959_bf99_ef542190ec08.slice/crio-a8bdf00e44a3cc9ff153e58147b8433a165b7f3717be7c99ed960560df3a0147 WatchSource:0}: Error finding container a8bdf00e44a3cc9ff153e58147b8433a165b7f3717be7c99ed960560df3a0147: Status 404 returned error can't find the container with id a8bdf00e44a3cc9ff153e58147b8433a165b7f3717be7c99ed960560df3a0147
Sep 15 19:15:44.904589 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.904553    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k" event={"ID":"e10ff7a4-0dbb-4959-bf99-ef542190ec08","Type":"ContainerStarted","Data":"f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1"}
Sep 15 19:15:44.905035 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.904603    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k" event={"ID":"e10ff7a4-0dbb-4959-bf99-ef542190ec08","Type":"ContainerStarted","Data":"a8bdf00e44a3cc9ff153e58147b8433a165b7f3717be7c99ed960560df3a0147"}
Sep 15 19:15:44.905035 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.904712    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:15:44.922661 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:44.922604    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k" podStartSLOduration=1.9225899 podStartE2EDuration="1.9225899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:44.921413369 +0000 UTC m=+1503.777266633" watchObservedRunningTime="2026-09-15 19:15:44.9225899 +0000 UTC m=+1503.778443211"
Sep 15 19:15:55.912320 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:15:55.912285    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:16:01.675349 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:01.675314    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:16:18.923960 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:18.923882    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"]
Sep 15 19:16:18.927706 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:18.927688    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:18.930154 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:18.930129    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-415f87\""
Sep 15 19:16:18.930275 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:18.930243    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-415f87-dockercfg-7lwnq\""
Sep 15 19:16:18.938118 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:18.938093    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"]
Sep 15 19:16:18.966321 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:18.966298    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbb9j\" (UniqueName: \"kubernetes.io/projected/1ae98ae1-ea22-4b15-909b-68cdc73b2972-kube-api-access-mbb9j\") pod \"payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6\" (UID: \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:18.966422 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:18.966334    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae98ae1-ea22-4b15-909b-68cdc73b2972-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6\" (UID: \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:19.067286 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:19.067247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbb9j\" (UniqueName: \"kubernetes.io/projected/1ae98ae1-ea22-4b15-909b-68cdc73b2972-kube-api-access-mbb9j\") pod \"payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6\" (UID: \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:19.067470 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:19.067295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae98ae1-ea22-4b15-909b-68cdc73b2972-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6\" (UID: \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:19.068086 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:19.068067    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae98ae1-ea22-4b15-909b-68cdc73b2972-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6\" (UID: \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:19.076651 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:19.076602    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbb9j\" (UniqueName: \"kubernetes.io/projected/1ae98ae1-ea22-4b15-909b-68cdc73b2972-kube-api-access-mbb9j\") pod \"payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6\" (UID: \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:19.239607 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:19.239527    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:19.514252 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:19.514219    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"]
Sep 15 19:16:19.516287 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:16:19.516261    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae98ae1_ea22_4b15_909b_68cdc73b2972.slice/crio-37ae875af2de0a425c73b53dc12d0cef0382df22aeefc1c7c6befb45f74cfe02 WatchSource:0}: Error finding container 37ae875af2de0a425c73b53dc12d0cef0382df22aeefc1c7c6befb45f74cfe02: Status 404 returned error can't find the container with id 37ae875af2de0a425c73b53dc12d0cef0382df22aeefc1c7c6befb45f74cfe02
Sep 15 19:16:20.109071 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:20.109026    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6" event={"ID":"1ae98ae1-ea22-4b15-909b-68cdc73b2972","Type":"ContainerStarted","Data":"e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2"}
Sep 15 19:16:20.109520 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:20.109092    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6" event={"ID":"1ae98ae1-ea22-4b15-909b-68cdc73b2972","Type":"ContainerStarted","Data":"37ae875af2de0a425c73b53dc12d0cef0382df22aeefc1c7c6befb45f74cfe02"}
Sep 15 19:16:20.109520 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:20.109124    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:16:20.127811 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:20.127758    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6" podStartSLOduration=2.127738556 podStartE2EDuration="2.127738556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:16:20.124950589 +0000 UTC m=+1538.980803858" watchObservedRunningTime="2026-09-15 19:16:20.127738556 +0000 UTC m=+1538.983591828"
Sep 15 19:16:31.117802 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:16:31.117769    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:17:12.672457 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:12.672421    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:17:17.672419 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:17.672378    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:17:43.491258 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:43.491180    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"]
Sep 15 19:17:43.504878 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:43.504848    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"]
Sep 15 19:17:43.510433 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:43.510387    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"]
Sep 15 19:17:49.258697 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.258659    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"]
Sep 15 19:17:49.259270 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.258918    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8" podUID="c15a9900-3226-4df5-b57c-62b5c2919583" containerName="payload-processing" containerID="cri-o://f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba" gracePeriod=30
Sep 15 19:17:49.514937 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.514876    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:17:49.575172 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.575142    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz2hc\" (UniqueName: \"kubernetes.io/projected/c15a9900-3226-4df5-b57c-62b5c2919583-kube-api-access-rz2hc\") pod \"c15a9900-3226-4df5-b57c-62b5c2919583\" (UID: \"c15a9900-3226-4df5-b57c-62b5c2919583\") "
Sep 15 19:17:49.575338 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.575198    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a9900-3226-4df5-b57c-62b5c2919583-plugins-config-volume\") pod \"c15a9900-3226-4df5-b57c-62b5c2919583\" (UID: \"c15a9900-3226-4df5-b57c-62b5c2919583\") "
Sep 15 19:17:49.575588 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.575565    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15a9900-3226-4df5-b57c-62b5c2919583-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c15a9900-3226-4df5-b57c-62b5c2919583" (UID: "c15a9900-3226-4df5-b57c-62b5c2919583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:49.575691 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.575671    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a9900-3226-4df5-b57c-62b5c2919583-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:49.577324 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.577297    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15a9900-3226-4df5-b57c-62b5c2919583-kube-api-access-rz2hc" (OuterVolumeSpecName: "kube-api-access-rz2hc") pod "c15a9900-3226-4df5-b57c-62b5c2919583" (UID: "c15a9900-3226-4df5-b57c-62b5c2919583"). InnerVolumeSpecName "kube-api-access-rz2hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:49.654318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.654238    2556 generic.go:356] "Generic (PLEG): container finished" podID="c15a9900-3226-4df5-b57c-62b5c2919583" containerID="f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba" exitCode=0
Sep 15 19:17:49.654318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.654292    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"
Sep 15 19:17:49.654318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.654300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8" event={"ID":"c15a9900-3226-4df5-b57c-62b5c2919583","Type":"ContainerDied","Data":"f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba"}
Sep 15 19:17:49.654624 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.654348    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8" event={"ID":"c15a9900-3226-4df5-b57c-62b5c2919583","Type":"ContainerDied","Data":"30fb3273a16485113f35472604ffaec1851c7582e01ef57a86c070f6716013a6"}
Sep 15 19:17:49.654624 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.654377    2556 scope.go:117] "RemoveContainer" containerID="f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba"
Sep 15 19:17:49.667530 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.667503    2556 scope.go:117] "RemoveContainer" containerID="f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba"
Sep 15 19:17:49.667941 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:17:49.667917    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba\": container with ID starting with f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba not found: ID does not exist" containerID="f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba"
Sep 15 19:17:49.668040 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.667952    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba"} err="failed to get container status \"f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba\": rpc error: code = NotFound desc = could not find container \"f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba\": container with ID starting with f107b031221f910eca6985321bd84c897c82b08329fc4a682e6439a652d887ba not found: ID does not exist"
Sep 15 19:17:49.676929 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.676908    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz2hc\" (UniqueName: \"kubernetes.io/projected/c15a9900-3226-4df5-b57c-62b5c2919583-kube-api-access-rz2hc\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:49.681502 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.681480    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"]
Sep 15 19:17:49.683597 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:49.683577    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17c562de-677f698857-ckph8"]
Sep 15 19:17:50.503417 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.503382    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"]
Sep 15 19:17:50.503906 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.503693    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt" podUID="6665de43-ff3f-4afa-a26b-899f208c7723" containerName="payload-pre-processing" containerID="cri-o://7ecd665a3be629a4f4fc2df52d7de55e0adc7ccf5bf4fb983768a1bcb361e799" gracePeriod=30
Sep 15 19:17:50.665393 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.665349    2556 generic.go:356] "Generic (PLEG): container finished" podID="6665de43-ff3f-4afa-a26b-899f208c7723" containerID="7ecd665a3be629a4f4fc2df52d7de55e0adc7ccf5bf4fb983768a1bcb361e799" exitCode=0
Sep 15 19:17:50.665570 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.665421    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt" event={"ID":"6665de43-ff3f-4afa-a26b-899f208c7723","Type":"ContainerDied","Data":"7ecd665a3be629a4f4fc2df52d7de55e0adc7ccf5bf4fb983768a1bcb361e799"}
Sep 15 19:17:50.772728 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.772702    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:17:50.891858 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.891831    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv7hh\" (UniqueName: \"kubernetes.io/projected/6665de43-ff3f-4afa-a26b-899f208c7723-kube-api-access-bv7hh\") pod \"6665de43-ff3f-4afa-a26b-899f208c7723\" (UID: \"6665de43-ff3f-4afa-a26b-899f208c7723\") "
Sep 15 19:17:50.892010 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.891891    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665de43-ff3f-4afa-a26b-899f208c7723-plugins-config-volume\") pod \"6665de43-ff3f-4afa-a26b-899f208c7723\" (UID: \"6665de43-ff3f-4afa-a26b-899f208c7723\") "
Sep 15 19:17:50.892266 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.892245    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6665de43-ff3f-4afa-a26b-899f208c7723-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6665de43-ff3f-4afa-a26b-899f208c7723" (UID: "6665de43-ff3f-4afa-a26b-899f208c7723"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:50.893867 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.893844    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6665de43-ff3f-4afa-a26b-899f208c7723-kube-api-access-bv7hh" (OuterVolumeSpecName: "kube-api-access-bv7hh") pod "6665de43-ff3f-4afa-a26b-899f208c7723" (UID: "6665de43-ff3f-4afa-a26b-899f208c7723"). InnerVolumeSpecName "kube-api-access-bv7hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:50.993771 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.993742    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv7hh\" (UniqueName: \"kubernetes.io/projected/6665de43-ff3f-4afa-a26b-899f208c7723-kube-api-access-bv7hh\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:50.993771 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:50.993767    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665de43-ff3f-4afa-a26b-899f208c7723-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:51.674570 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:51.674533    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"
Sep 15 19:17:51.679180 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:51.679150    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15a9900-3226-4df5-b57c-62b5c2919583" path="/var/lib/kubelet/pods/c15a9900-3226-4df5-b57c-62b5c2919583/volumes"
Sep 15 19:17:51.679544 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:51.679527    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt" event={"ID":"6665de43-ff3f-4afa-a26b-899f208c7723","Type":"ContainerDied","Data":"7c41f0bbd687428f2e0661d0482927db14f42187c3bab76543eb877cd597b8b8"}
Sep 15 19:17:51.679616 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:51.679570    2556 scope.go:117] "RemoveContainer" containerID="7ecd665a3be629a4f4fc2df52d7de55e0adc7ccf5bf4fb983768a1bcb361e799"
Sep 15 19:17:51.705680 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:51.705630    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"]
Sep 15 19:17:51.707819 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:51.707797    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f0ad35-7f597bc94b-qvkbt"]
Sep 15 19:17:53.680581 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:53.680549    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6665de43-ff3f-4afa-a26b-899f208c7723" path="/var/lib/kubelet/pods/6665de43-ff3f-4afa-a26b-899f208c7723/volumes"
Sep 15 19:17:58.511693 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.511632    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd" podUID="f004b09b-d718-44d2-a4a9-91977925ed3b" containerName="main" containerID="cri-o://01be4caf2fe743a46432f8a882e4796143c60f283f1ce523e519b0183cf0e73e" gracePeriod=45
Sep 15 19:17:58.524937 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.524903    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl" podUID="55635747-71db-4cb1-b0b2-c6b07f683278" containerName="main" containerID="cri-o://f7eebb881906a7ccdc3d8db454a4331e0df8dd0c6bb121298c9aa7346f44b9b9" gracePeriod=45
Sep 15 19:17:58.529545 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.529516    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz" podUID="5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" containerName="main" containerID="cri-o://add4fed4b0b21bff2cf6b1b203b2167839bc9c43dfb1e08e3b90ae45992a1b21" gracePeriod=45
Sep 15 19:17:58.713537 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.713508    2556 generic.go:356] "Generic (PLEG): container finished" podID="f004b09b-d718-44d2-a4a9-91977925ed3b" containerID="01be4caf2fe743a46432f8a882e4796143c60f283f1ce523e519b0183cf0e73e" exitCode=0
Sep 15 19:17:58.713711 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.713578    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd" event={"ID":"f004b09b-d718-44d2-a4a9-91977925ed3b","Type":"ContainerDied","Data":"01be4caf2fe743a46432f8a882e4796143c60f283f1ce523e519b0183cf0e73e"}
Sep 15 19:17:58.715261 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.715237    2556 generic.go:356] "Generic (PLEG): container finished" podID="5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" containerID="add4fed4b0b21bff2cf6b1b203b2167839bc9c43dfb1e08e3b90ae45992a1b21" exitCode=0
Sep 15 19:17:58.715352 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.715305    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz" event={"ID":"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e","Type":"ContainerDied","Data":"add4fed4b0b21bff2cf6b1b203b2167839bc9c43dfb1e08e3b90ae45992a1b21"}
Sep 15 19:17:58.716945 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.716927    2556 generic.go:356] "Generic (PLEG): container finished" podID="55635747-71db-4cb1-b0b2-c6b07f683278" containerID="f7eebb881906a7ccdc3d8db454a4331e0df8dd0c6bb121298c9aa7346f44b9b9" exitCode=0
Sep 15 19:17:58.717026 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.716997    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl" event={"ID":"55635747-71db-4cb1-b0b2-c6b07f683278","Type":"ContainerDied","Data":"f7eebb881906a7ccdc3d8db454a4331e0df8dd0c6bb121298c9aa7346f44b9b9"}
Sep 15 19:17:58.818222 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.818192    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:17:58.843409 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.843381    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:17:58.847818 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.847791    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:17:58.976176 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976143    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55635747-71db-4cb1-b0b2-c6b07f683278-tls-certs\") pod \"55635747-71db-4cb1-b0b2-c6b07f683278\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") "
Sep 15 19:17:58.976349 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976194    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-dshm\") pod \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") "
Sep 15 19:17:58.976349 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976246    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-dshm\") pod \"55635747-71db-4cb1-b0b2-c6b07f683278\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") "
Sep 15 19:17:58.976349 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976272    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhzdk\" (UniqueName: \"kubernetes.io/projected/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-kube-api-access-jhzdk\") pod \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") "
Sep 15 19:17:58.976349 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976308    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f004b09b-d718-44d2-a4a9-91977925ed3b-tls-certs\") pod \"f004b09b-d718-44d2-a4a9-91977925ed3b\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") "
Sep 15 19:17:58.976349 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976345    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9r5g\" (UniqueName: \"kubernetes.io/projected/55635747-71db-4cb1-b0b2-c6b07f683278-kube-api-access-t9r5g\") pod \"55635747-71db-4cb1-b0b2-c6b07f683278\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") "
Sep 15 19:17:58.976633 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976377    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-model-cache\") pod \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") "
Sep 15 19:17:58.976633 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976442    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-model-cache\") pod \"f004b09b-d718-44d2-a4a9-91977925ed3b\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") "
Sep 15 19:17:58.976774 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976743    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f004b09b-d718-44d2-a4a9-91977925ed3b" (UID: "f004b09b-d718-44d2-a4a9-91977925ed3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.976828 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976758    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" (UID: "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.976884 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976863    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-home\") pod \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") "
Sep 15 19:17:58.976936 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976910    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tmp-dir\") pod \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") "
Sep 15 19:17:58.976992 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976964    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-home\") pod \"f004b09b-d718-44d2-a4a9-91977925ed3b\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") "
Sep 15 19:17:58.977048 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.976990    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-tmp-dir\") pod \"f004b09b-d718-44d2-a4a9-91977925ed3b\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") "
Sep 15 19:17:58.977048 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977018    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdw7h\" (UniqueName: \"kubernetes.io/projected/f004b09b-d718-44d2-a4a9-91977925ed3b-kube-api-access-pdw7h\") pod \"f004b09b-d718-44d2-a4a9-91977925ed3b\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") "
Sep 15 19:17:58.977155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977047    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-home\") pod \"55635747-71db-4cb1-b0b2-c6b07f683278\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") "
Sep 15 19:17:58.977155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977099    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-home" (OuterVolumeSpecName: "home") pod "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" (UID: "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.977155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977136    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tls-certs\") pod \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\" (UID: \"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e\") "
Sep 15 19:17:58.977312 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977161    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-tmp-dir\") pod \"55635747-71db-4cb1-b0b2-c6b07f683278\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") "
Sep 15 19:17:58.977312 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977188    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-model-cache\") pod \"55635747-71db-4cb1-b0b2-c6b07f683278\" (UID: \"55635747-71db-4cb1-b0b2-c6b07f683278\") "
Sep 15 19:17:58.977312 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977221    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-dshm\") pod \"f004b09b-d718-44d2-a4a9-91977925ed3b\" (UID: \"f004b09b-d718-44d2-a4a9-91977925ed3b\") "
Sep 15 19:17:58.977312 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977259    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-home" (OuterVolumeSpecName: "home") pod "f004b09b-d718-44d2-a4a9-91977925ed3b" (UID: "f004b09b-d718-44d2-a4a9-91977925ed3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.977564 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977395    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" (UID: "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.977564 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977542    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f004b09b-d718-44d2-a4a9-91977925ed3b" (UID: "f004b09b-d718-44d2-a4a9-91977925ed3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.977974 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977745    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-model-cache\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:58.977974 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977776    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-home\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:58.977974 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977794    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tmp-dir\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:58.977974 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977810    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-home\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:58.977974 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977828    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-tmp-dir\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:58.977974 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977885    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-model-cache\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:58.978369 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.978099    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-model-cache" (OuterVolumeSpecName: "model-cache") pod "55635747-71db-4cb1-b0b2-c6b07f683278" (UID: "55635747-71db-4cb1-b0b2-c6b07f683278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.978369 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.977555    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-home" (OuterVolumeSpecName: "home") pod "55635747-71db-4cb1-b0b2-c6b07f683278" (UID: "55635747-71db-4cb1-b0b2-c6b07f683278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.978369 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.978257    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55635747-71db-4cb1-b0b2-c6b07f683278" (UID: "55635747-71db-4cb1-b0b2-c6b07f683278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.979237 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.979202    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f004b09b-d718-44d2-a4a9-91977925ed3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f004b09b-d718-44d2-a4a9-91977925ed3b" (UID: "f004b09b-d718-44d2-a4a9-91977925ed3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:58.979237 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.979226    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55635747-71db-4cb1-b0b2-c6b07f683278-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55635747-71db-4cb1-b0b2-c6b07f683278" (UID: "55635747-71db-4cb1-b0b2-c6b07f683278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:58.979507 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.979481    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55635747-71db-4cb1-b0b2-c6b07f683278-kube-api-access-t9r5g" (OuterVolumeSpecName: "kube-api-access-t9r5g") pod "55635747-71db-4cb1-b0b2-c6b07f683278" (UID: "55635747-71db-4cb1-b0b2-c6b07f683278"). InnerVolumeSpecName "kube-api-access-t9r5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:58.979839 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.979814    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-kube-api-access-jhzdk" (OuterVolumeSpecName: "kube-api-access-jhzdk") pod "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" (UID: "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e"). InnerVolumeSpecName "kube-api-access-jhzdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:58.979966 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.979943    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-dshm" (OuterVolumeSpecName: "dshm") pod "55635747-71db-4cb1-b0b2-c6b07f683278" (UID: "55635747-71db-4cb1-b0b2-c6b07f683278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.980235 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.980217    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-dshm" (OuterVolumeSpecName: "dshm") pod "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" (UID: "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.980843 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.980826    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004b09b-d718-44d2-a4a9-91977925ed3b-kube-api-access-pdw7h" (OuterVolumeSpecName: "kube-api-access-pdw7h") pod "f004b09b-d718-44d2-a4a9-91977925ed3b" (UID: "f004b09b-d718-44d2-a4a9-91977925ed3b"). InnerVolumeSpecName "kube-api-access-pdw7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:58.980933 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.980909    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" (UID: "5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:58.981407 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:58.981380    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-dshm" (OuterVolumeSpecName: "dshm") pod "f004b09b-d718-44d2-a4a9-91977925ed3b" (UID: "f004b09b-d718-44d2-a4a9-91977925ed3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079116    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhzdk\" (UniqueName: \"kubernetes.io/projected/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-kube-api-access-jhzdk\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079149    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f004b09b-d718-44d2-a4a9-91977925ed3b-tls-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079160    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9r5g\" (UniqueName: \"kubernetes.io/projected/55635747-71db-4cb1-b0b2-c6b07f683278-kube-api-access-t9r5g\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079169    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdw7h\" (UniqueName: \"kubernetes.io/projected/f004b09b-d718-44d2-a4a9-91977925ed3b-kube-api-access-pdw7h\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079180    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-home\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079188    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-tls-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079197    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-tmp-dir\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079204    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-model-cache\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079206 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079213    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f004b09b-d718-44d2-a4a9-91977925ed3b-dshm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079597 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079219    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55635747-71db-4cb1-b0b2-c6b07f683278-tls-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079597 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079226    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e-dshm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.079597 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.079233    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55635747-71db-4cb1-b0b2-c6b07f683278-dshm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.723694 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.723629    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"
Sep 15 19:17:59.724061 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.723627    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd" event={"ID":"f004b09b-d718-44d2-a4a9-91977925ed3b","Type":"ContainerDied","Data":"07af0f165bcf6ac4156ddded2a5205fcca9f39c5c685976e7168d95420fa4de9"}
Sep 15 19:17:59.724061 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.723799    2556 scope.go:117] "RemoveContainer" containerID="01be4caf2fe743a46432f8a882e4796143c60f283f1ce523e519b0183cf0e73e"
Sep 15 19:17:59.725701 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.725674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz" event={"ID":"5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e","Type":"ContainerDied","Data":"d49587eddf6da11968ac022e6a74a0d5c2f1b085702fc477053dfa730cd58f20"}
Sep 15 19:17:59.725810 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.725701    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"
Sep 15 19:17:59.727324 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.727301    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl" event={"ID":"55635747-71db-4cb1-b0b2-c6b07f683278","Type":"ContainerDied","Data":"82b02d4911a54c04f606ce6ee52f2cef0e50c56384749b62707f6bdeed2cac31"}
Sep 15 19:17:59.727439 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.727358    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"
Sep 15 19:17:59.734233 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.734214    2556 scope.go:117] "RemoveContainer" containerID="add4fed4b0b21bff2cf6b1b203b2167839bc9c43dfb1e08e3b90ae45992a1b21"
Sep 15 19:17:59.742527 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.742507    2556 scope.go:117] "RemoveContainer" containerID="f7eebb881906a7ccdc3d8db454a4331e0df8dd0c6bb121298c9aa7346f44b9b9"
Sep 15 19:17:59.743884 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.743863    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"]
Sep 15 19:17:59.750125 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.750103    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-w3-f0ad35-kserve-7555d5778c-5qfdd"]
Sep 15 19:17:59.759134 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.759109    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"]
Sep 15 19:17:59.764672 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.764619    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/premium-w3-f0ad35-kserve-748b7d4848-kh6zl"]
Sep 15 19:17:59.773749 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.773730    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"]
Sep 15 19:17:59.777089 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:17:59.777040    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/simulator-w3-f0ad35-kserve-67d549dd85-bw4jz"]
Sep 15 19:18:01.678479 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:01.678441    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55635747-71db-4cb1-b0b2-c6b07f683278" path="/var/lib/kubelet/pods/55635747-71db-4cb1-b0b2-c6b07f683278/volumes"
Sep 15 19:18:01.678912 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:01.678897    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" path="/var/lib/kubelet/pods/5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e/volumes"
Sep 15 19:18:01.679260 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:01.679248    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f004b09b-d718-44d2-a4a9-91977925ed3b" path="/var/lib/kubelet/pods/f004b09b-d718-44d2-a4a9-91977925ed3b/volumes"
Sep 15 19:18:11.464120 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464082    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"]
Sep 15 19:18:11.464840 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464819    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" containerName="main"
Sep 15 19:18:11.464897 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464845    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" containerName="main"
Sep 15 19:18:11.464897 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464883    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15a9900-3226-4df5-b57c-62b5c2919583" containerName="payload-processing"
Sep 15 19:18:11.464897 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464893    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15a9900-3226-4df5-b57c-62b5c2919583" containerName="payload-processing"
Sep 15 19:18:11.464997 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464907    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6665de43-ff3f-4afa-a26b-899f208c7723" containerName="payload-pre-processing"
Sep 15 19:18:11.464997 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464915    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6665de43-ff3f-4afa-a26b-899f208c7723" containerName="payload-pre-processing"
Sep 15 19:18:11.464997 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464931    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55635747-71db-4cb1-b0b2-c6b07f683278" containerName="main"
Sep 15 19:18:11.464997 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464939    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="55635747-71db-4cb1-b0b2-c6b07f683278" containerName="main"
Sep 15 19:18:11.464997 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464954    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f004b09b-d718-44d2-a4a9-91977925ed3b" containerName="main"
Sep 15 19:18:11.464997 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.464962    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f004b09b-d718-44d2-a4a9-91977925ed3b" containerName="main"
Sep 15 19:18:11.465171 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.465067    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf02fcb-e9dc-448a-a3d5-b3f511de7b1e" containerName="main"
Sep 15 19:18:11.465171 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.465080    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f004b09b-d718-44d2-a4a9-91977925ed3b" containerName="main"
Sep 15 19:18:11.465171 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.465092    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6665de43-ff3f-4afa-a26b-899f208c7723" containerName="payload-pre-processing"
Sep 15 19:18:11.465171 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.465108    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="55635747-71db-4cb1-b0b2-c6b07f683278" containerName="main"
Sep 15 19:18:11.465171 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.465117    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15a9900-3226-4df5-b57c-62b5c2919583" containerName="payload-processing"
Sep 15 19:18:11.468867 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.468847    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.472855 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.472700    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"auth-test-model-415f87-kserve-self-signed-certs\""
Sep 15 19:18:11.472855 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.472770    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"openshift-service-ca.crt\""
Sep 15 19:18:11.473083 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.472721    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"kube-root-ca.crt\""
Sep 15 19:18:11.473879 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.473856    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"default-dockercfg-j4qsp\""
Sep 15 19:18:11.476901 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.476879    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"]
Sep 15 19:18:11.502906 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.502874    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-model-cache\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.503021 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.502944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-home\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.503021 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.502999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lthf9\" (UniqueName: \"kubernetes.io/projected/aba7ed5c-4512-4ab5-be20-80608659f886-kube-api-access-lthf9\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.503021 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.503017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba7ed5c-4512-4ab5-be20-80608659f886-tls-certs\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.503155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.503037    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-dshm\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.503155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.503063    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-tmp-dir\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604036 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604004    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthf9\" (UniqueName: \"kubernetes.io/projected/aba7ed5c-4512-4ab5-be20-80608659f886-kube-api-access-lthf9\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604196 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604041    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba7ed5c-4512-4ab5-be20-80608659f886-tls-certs\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604196 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604063    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-dshm\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604319 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-tmp-dir\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604319 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604275    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-model-cache\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604422 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604388    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-home\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604614 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604592    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-tmp-dir\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604720 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604593    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-model-cache\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.604790 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.604729    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-home\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.606865 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.606835    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-dshm\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.607120 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.607103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba7ed5c-4512-4ab5-be20-80608659f886-tls-certs\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.612387 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.612365    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthf9\" (UniqueName: \"kubernetes.io/projected/aba7ed5c-4512-4ab5-be20-80608659f886-kube-api-access-lthf9\") pod \"auth-test-model-415f87-kserve-98dbc7d79-sdsm2\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") " pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:11.780620 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:11.780521    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:12.053240 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:12.053211    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"]
Sep 15 19:18:12.054966 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:18:12.054935    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaba7ed5c_4512_4ab5_be20_80608659f886.slice/crio-124d486dc47e38a8aa1756937fd072e4711fc332e56d9fa19bcf60c62a1d3dbc WatchSource:0}: Error finding container 124d486dc47e38a8aa1756937fd072e4711fc332e56d9fa19bcf60c62a1d3dbc: Status 404 returned error can't find the container with id 124d486dc47e38a8aa1756937fd072e4711fc332e56d9fa19bcf60c62a1d3dbc
Sep 15 19:18:12.807127 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:12.807092    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2" event={"ID":"aba7ed5c-4512-4ab5-be20-80608659f886","Type":"ContainerStarted","Data":"bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5"}
Sep 15 19:18:12.807497 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:12.807136    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2" event={"ID":"aba7ed5c-4512-4ab5-be20-80608659f886","Type":"ContainerStarted","Data":"124d486dc47e38a8aa1756937fd072e4711fc332e56d9fa19bcf60c62a1d3dbc"}
Sep 15 19:18:12.825559 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:12.825515    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2" podStartSLOduration=1.825498354 podStartE2EDuration="1.825498354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:18:12.824204889 +0000 UTC m=+1651.680058169" watchObservedRunningTime="2026-09-15 19:18:12.825498354 +0000 UTC m=+1651.681351622"
Sep 15 19:18:21.781201 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:21.781168    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:21.781712 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:21.781221    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:21.793848 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:21.793819    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:21.867177 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:21.867146    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:18:28.672394 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:28.672355    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:18:47.672410 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:18:47.672375    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:19:02.783299 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.783263    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"]
Sep 15 19:19:02.786912 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.786896    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:02.789228 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.789206    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"embedding-w0-900c88-kserve-self-signed-certs\""
Sep 15 19:19:02.797461 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.797433    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"]
Sep 15 19:19:02.899150 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.899117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-home\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:02.899150 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.899153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-dshm\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:02.899348 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.899190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-model-cache\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:02.899348 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.899255    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-tmp-dir\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:02.899348 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.899299    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93351b1-967d-4f4d-9eb7-f41756ad3241-tls-certs\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:02.899450 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:02.899354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twwhf\" (UniqueName: \"kubernetes.io/projected/d93351b1-967d-4f4d-9eb7-f41756ad3241-kube-api-access-twwhf\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.000338 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000306    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-tmp-dir\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.000532 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93351b1-967d-4f4d-9eb7-f41756ad3241-tls-certs\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.000532 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000424    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twwhf\" (UniqueName: \"kubernetes.io/projected/d93351b1-967d-4f4d-9eb7-f41756ad3241-kube-api-access-twwhf\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.000532 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000457    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-home\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.000532 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-dshm\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.000792 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-model-cache\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.000792 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-tmp-dir\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.000906 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000804    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-home\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.001004 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.000985    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-model-cache\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.003141 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.003110    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-dshm\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.003274 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.003184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93351b1-967d-4f4d-9eb7-f41756ad3241-tls-certs\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.010658 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.008624    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twwhf\" (UniqueName: \"kubernetes.io/projected/d93351b1-967d-4f4d-9eb7-f41756ad3241-kube-api-access-twwhf\") pod \"embedding-w0-900c88-kserve-86b8df66b8-hx9pz\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") " pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.098093 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.098061    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:03.369214 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:03.369181    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"]
Sep 15 19:19:03.370809 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:19:03.370779    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd93351b1_967d_4f4d_9eb7_f41756ad3241.slice/crio-1ace36921e9c98942b105a3f95bb028d8ae3ac5f3825ad55d6b664329228bd7a WatchSource:0}: Error finding container 1ace36921e9c98942b105a3f95bb028d8ae3ac5f3825ad55d6b664329228bd7a: Status 404 returned error can't find the container with id 1ace36921e9c98942b105a3f95bb028d8ae3ac5f3825ad55d6b664329228bd7a
Sep 15 19:19:04.088180 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:04.088090    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz" event={"ID":"d93351b1-967d-4f4d-9eb7-f41756ad3241","Type":"ContainerStarted","Data":"52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae"}
Sep 15 19:19:04.088180 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:04.088143    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz" event={"ID":"d93351b1-967d-4f4d-9eb7-f41756ad3241","Type":"ContainerStarted","Data":"1ace36921e9c98942b105a3f95bb028d8ae3ac5f3825ad55d6b664329228bd7a"}
Sep 15 19:19:04.102973 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:04.102924    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz" podStartSLOduration=2.102909016 podStartE2EDuration="2.102909016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:04.102025274 +0000 UTC m=+1702.957878543" watchObservedRunningTime="2026-09-15 19:19:04.102909016 +0000 UTC m=+1702.958762428"
Sep 15 19:19:13.098205 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:13.098168    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:13.098598 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:13.098220    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:13.111368 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:13.111337    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:13.148396 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:13.148371    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:19:36.672340 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:36.672305    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:19:47.411286 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.411243    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"]
Sep 15 19:19:47.415104 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.415081    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:47.417444 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.417420    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-759580-dockercfg-w4v8q\""
Sep 15 19:19:47.417552 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.417426    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-759580\""
Sep 15 19:19:47.422630 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.422602    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"]
Sep 15 19:19:47.449495 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.449467    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"]
Sep 15 19:19:47.453095 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.453078    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:47.462214 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.462191    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"]
Sep 15 19:19:47.534910 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.534875    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbbx\" (UniqueName: \"kubernetes.io/projected/68dc6f53-9def-4871-99d2-cb691736750f-kube-api-access-jhbbx\") pod \"payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b\" (UID: \"68dc6f53-9def-4871-99d2-cb691736750f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:47.535077 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.535050    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dc6f53-9def-4871-99d2-cb691736750f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b\" (UID: \"68dc6f53-9def-4871-99d2-cb691736750f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:47.636360 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.636308    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744jb\" (UniqueName: \"kubernetes.io/projected/bc0df392-dde3-421c-a312-b4ee50ab3ad6-kube-api-access-744jb\") pod \"payload-processing-e2e-api-a-759580-74684d8d79-fdx64\" (UID: \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\") " pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:47.636360 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.636365    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dc6f53-9def-4871-99d2-cb691736750f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b\" (UID: \"68dc6f53-9def-4871-99d2-cb691736750f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:47.636579 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.636486    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbbx\" (UniqueName: \"kubernetes.io/projected/68dc6f53-9def-4871-99d2-cb691736750f-kube-api-access-jhbbx\") pod \"payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b\" (UID: \"68dc6f53-9def-4871-99d2-cb691736750f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:47.636617 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.636598    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0df392-dde3-421c-a312-b4ee50ab3ad6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-759580-74684d8d79-fdx64\" (UID: \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\") " pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:47.637177 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.637157    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dc6f53-9def-4871-99d2-cb691736750f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b\" (UID: \"68dc6f53-9def-4871-99d2-cb691736750f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:47.645226 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.645204    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbbx\" (UniqueName: \"kubernetes.io/projected/68dc6f53-9def-4871-99d2-cb691736750f-kube-api-access-jhbbx\") pod \"payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b\" (UID: \"68dc6f53-9def-4871-99d2-cb691736750f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:47.729220 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.729157    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:47.737865 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.737839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0df392-dde3-421c-a312-b4ee50ab3ad6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-759580-74684d8d79-fdx64\" (UID: \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\") " pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:47.737988 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.737898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744jb\" (UniqueName: \"kubernetes.io/projected/bc0df392-dde3-421c-a312-b4ee50ab3ad6-kube-api-access-744jb\") pod \"payload-processing-e2e-api-a-759580-74684d8d79-fdx64\" (UID: \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\") " pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:47.738586 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.738568    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0df392-dde3-421c-a312-b4ee50ab3ad6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-759580-74684d8d79-fdx64\" (UID: \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\") " pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:47.745837 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.745818    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-744jb\" (UniqueName: \"kubernetes.io/projected/bc0df392-dde3-421c-a312-b4ee50ab3ad6-kube-api-access-744jb\") pod \"payload-processing-e2e-api-a-759580-74684d8d79-fdx64\" (UID: \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\") " pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:47.766525 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:47.766491    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:48.002632 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.002608    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"]
Sep 15 19:19:48.004199 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:19:48.004171    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dc6f53_9def_4871_99d2_cb691736750f.slice/crio-797632658b1bd10536741d6943a96fbc3ffd5bc06598deb34fff034ee0ab7135 WatchSource:0}: Error finding container 797632658b1bd10536741d6943a96fbc3ffd5bc06598deb34fff034ee0ab7135: Status 404 returned error can't find the container with id 797632658b1bd10536741d6943a96fbc3ffd5bc06598deb34fff034ee0ab7135
Sep 15 19:19:48.059361 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.059333    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"]
Sep 15 19:19:48.059752 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:19:48.059723    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0df392_dde3_421c_a312_b4ee50ab3ad6.slice/crio-9746c549b76ac47057a5b8fffb9a30284208eeec5da07af21acebd89b7fd3d64 WatchSource:0}: Error finding container 9746c549b76ac47057a5b8fffb9a30284208eeec5da07af21acebd89b7fd3d64: Status 404 returned error can't find the container with id 9746c549b76ac47057a5b8fffb9a30284208eeec5da07af21acebd89b7fd3d64
Sep 15 19:19:48.331230 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.331129    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64" event={"ID":"bc0df392-dde3-421c-a312-b4ee50ab3ad6","Type":"ContainerStarted","Data":"df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407"}
Sep 15 19:19:48.331230 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.331203    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:48.331455 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.331229    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64" event={"ID":"bc0df392-dde3-421c-a312-b4ee50ab3ad6","Type":"ContainerStarted","Data":"9746c549b76ac47057a5b8fffb9a30284208eeec5da07af21acebd89b7fd3d64"}
Sep 15 19:19:48.332843 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.332816    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b" event={"ID":"68dc6f53-9def-4871-99d2-cb691736750f","Type":"ContainerStarted","Data":"f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0"}
Sep 15 19:19:48.332954 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.332857    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b" event={"ID":"68dc6f53-9def-4871-99d2-cb691736750f","Type":"ContainerStarted","Data":"797632658b1bd10536741d6943a96fbc3ffd5bc06598deb34fff034ee0ab7135"}
Sep 15 19:19:48.332954 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.332876    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:19:48.354288 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.354238    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64" podStartSLOduration=1.354221967 podStartE2EDuration="1.354221967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:48.353796804 +0000 UTC m=+1747.209650073" watchObservedRunningTime="2026-09-15 19:19:48.354221967 +0000 UTC m=+1747.210075239"
Sep 15 19:19:48.373495 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:48.373442    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b" podStartSLOduration=1.373424426 podStartE2EDuration="1.373424426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:48.372772892 +0000 UTC m=+1747.228626185" watchObservedRunningTime="2026-09-15 19:19:48.373424426 +0000 UTC m=+1747.229277695"
Sep 15 19:19:59.343573 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:59.343537    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:19:59.344079 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:19:59.343676    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:20:01.376565 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:01.376529    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"]
Sep 15 19:20:01.386767 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:01.386723    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"]
Sep 15 19:20:03.672378 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:03.672343    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:20:12.543474 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:12.543439    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"]
Sep 15 19:20:12.544070 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:12.543873    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg" podUID="83fcaeca-caa2-46f1-8b42-922e878ed390" containerName="maas-api" containerID="cri-o://48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc" gracePeriod=30
Sep 15 19:20:12.574121 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:12.574089    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"]
Sep 15 19:20:12.576047 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:12.576024    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-900c88-29824995-bgs22"]
Sep 15 19:20:12.999631 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:12.999607    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:20:13.093971 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.093894    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-metrics-certs\") pod \"83fcaeca-caa2-46f1-8b42-922e878ed390\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") "
Sep 15 19:20:13.094131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.093990    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls\") pod \"83fcaeca-caa2-46f1-8b42-922e878ed390\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") "
Sep 15 19:20:13.094131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.094052    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kxph\" (UniqueName: \"kubernetes.io/projected/83fcaeca-caa2-46f1-8b42-922e878ed390-kube-api-access-9kxph\") pod \"83fcaeca-caa2-46f1-8b42-922e878ed390\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") "
Sep 15 19:20:13.094131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.094079    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcaeca-caa2-46f1-8b42-922e878ed390-openshift-service-ca\") pod \"83fcaeca-caa2-46f1-8b42-922e878ed390\" (UID: \"83fcaeca-caa2-46f1-8b42-922e878ed390\") "
Sep 15 19:20:13.094495 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.094432    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fcaeca-caa2-46f1-8b42-922e878ed390-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83fcaeca-caa2-46f1-8b42-922e878ed390" (UID: "83fcaeca-caa2-46f1-8b42-922e878ed390"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:13.096355 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.096330    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fcaeca-caa2-46f1-8b42-922e878ed390-kube-api-access-9kxph" (OuterVolumeSpecName: "kube-api-access-9kxph") pod "83fcaeca-caa2-46f1-8b42-922e878ed390" (UID: "83fcaeca-caa2-46f1-8b42-922e878ed390"). InnerVolumeSpecName "kube-api-access-9kxph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:13.096355 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.096332    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "83fcaeca-caa2-46f1-8b42-922e878ed390" (UID: "83fcaeca-caa2-46f1-8b42-922e878ed390"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:13.096684 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.096663    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83fcaeca-caa2-46f1-8b42-922e878ed390" (UID: "83fcaeca-caa2-46f1-8b42-922e878ed390"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:13.195191 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.195158    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-maas-api-tls\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:13.195191 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.195184    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kxph\" (UniqueName: \"kubernetes.io/projected/83fcaeca-caa2-46f1-8b42-922e878ed390-kube-api-access-9kxph\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:13.195191 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.195193    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcaeca-caa2-46f1-8b42-922e878ed390-openshift-service-ca\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:13.195402 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.195202    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcaeca-caa2-46f1-8b42-922e878ed390-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:13.474343 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.474307    2556 generic.go:356] "Generic (PLEG): container finished" podID="83fcaeca-caa2-46f1-8b42-922e878ed390" containerID="48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc" exitCode=0
Sep 15 19:20:13.474517 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.474355    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"
Sep 15 19:20:13.474517 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.474403    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg" event={"ID":"83fcaeca-caa2-46f1-8b42-922e878ed390","Type":"ContainerDied","Data":"48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc"}
Sep 15 19:20:13.474517 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.474462    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg" event={"ID":"83fcaeca-caa2-46f1-8b42-922e878ed390","Type":"ContainerDied","Data":"58d71fb5c7eea04735405b867f77c3959f244f41ce147b0aa35d283ebf234744"}
Sep 15 19:20:13.474517 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.474484    2556 scope.go:117] "RemoveContainer" containerID="48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc"
Sep 15 19:20:13.484779 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.484761    2556 scope.go:117] "RemoveContainer" containerID="48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc"
Sep 15 19:20:13.485053 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:20:13.485035    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc\": container with ID starting with 48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc not found: ID does not exist" containerID="48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc"
Sep 15 19:20:13.485121 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.485063    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc"} err="failed to get container status \"48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc\": rpc error: code = NotFound desc = could not find container \"48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc\": container with ID starting with 48368ee717453209dc5a7a6e43c93096d1ad5bbf3d61bc71b6ba7b6e2e96c4cc not found: ID does not exist"
Sep 15 19:20:13.497491 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.497467    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"]
Sep 15 19:20:13.501141 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.501121    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-900c88-7964c5489d-46mbg"]
Sep 15 19:20:13.678709 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.678677    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83fcaeca-caa2-46f1-8b42-922e878ed390" path="/var/lib/kubelet/pods/83fcaeca-caa2-46f1-8b42-922e878ed390/volumes"
Sep 15 19:20:13.679093 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:13.679044    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12fc6c6-51e9-4d5e-bd77-40429e1edee4" path="/var/lib/kubelet/pods/d12fc6c6-51e9-4d5e-bd77-40429e1edee4/volumes"
Sep 15 19:20:16.397187 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.397103    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65" podUID="9ab597c1-ab83-411c-91b1-285d384a64da" containerName="main" containerID="cri-o://d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca" gracePeriod=45
Sep 15 19:20:16.407415 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.407351    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz" podUID="d93351b1-967d-4f4d-9eb7-f41756ad3241" containerName="main" containerID="cri-o://52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae" gracePeriod=45
Sep 15 19:20:16.657012 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.656951    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:20:16.688396 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.688374    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:20:16.733271 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733235    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-tmp-dir\") pod \"9ab597c1-ab83-411c-91b1-285d384a64da\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") "
Sep 15 19:20:16.733466 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733295    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-model-cache\") pod \"9ab597c1-ab83-411c-91b1-285d384a64da\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") "
Sep 15 19:20:16.733466 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733341    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-home\") pod \"9ab597c1-ab83-411c-91b1-285d384a64da\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") "
Sep 15 19:20:16.733466 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733382    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55bgs\" (UniqueName: \"kubernetes.io/projected/9ab597c1-ab83-411c-91b1-285d384a64da-kube-api-access-55bgs\") pod \"9ab597c1-ab83-411c-91b1-285d384a64da\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") "
Sep 15 19:20:16.733659 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733475    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab597c1-ab83-411c-91b1-285d384a64da-tls-certs\") pod \"9ab597c1-ab83-411c-91b1-285d384a64da\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") "
Sep 15 19:20:16.733659 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733503    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-dshm\") pod \"9ab597c1-ab83-411c-91b1-285d384a64da\" (UID: \"9ab597c1-ab83-411c-91b1-285d384a64da\") "
Sep 15 19:20:16.733659 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733562    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ab597c1-ab83-411c-91b1-285d384a64da" (UID: "9ab597c1-ab83-411c-91b1-285d384a64da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.733800 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733725    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-home" (OuterVolumeSpecName: "home") pod "9ab597c1-ab83-411c-91b1-285d384a64da" (UID: "9ab597c1-ab83-411c-91b1-285d384a64da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.733950 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733926    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-tmp-dir\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.734017 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733956    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-home\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.734017 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.733964    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ab597c1-ab83-411c-91b1-285d384a64da" (UID: "9ab597c1-ab83-411c-91b1-285d384a64da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.736224 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.736197    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab597c1-ab83-411c-91b1-285d384a64da-kube-api-access-55bgs" (OuterVolumeSpecName: "kube-api-access-55bgs") pod "9ab597c1-ab83-411c-91b1-285d384a64da" (UID: "9ab597c1-ab83-411c-91b1-285d384a64da"). InnerVolumeSpecName "kube-api-access-55bgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:16.736374 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.736353    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab597c1-ab83-411c-91b1-285d384a64da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ab597c1-ab83-411c-91b1-285d384a64da" (UID: "9ab597c1-ab83-411c-91b1-285d384a64da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:16.736454 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.736358    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-dshm" (OuterVolumeSpecName: "dshm") pod "9ab597c1-ab83-411c-91b1-285d384a64da" (UID: "9ab597c1-ab83-411c-91b1-285d384a64da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.834902 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.834859    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twwhf\" (UniqueName: \"kubernetes.io/projected/d93351b1-967d-4f4d-9eb7-f41756ad3241-kube-api-access-twwhf\") pod \"d93351b1-967d-4f4d-9eb7-f41756ad3241\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") "
Sep 15 19:20:16.835055 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.834939    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-model-cache\") pod \"d93351b1-967d-4f4d-9eb7-f41756ad3241\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") "
Sep 15 19:20:16.835055 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.834993    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-dshm\") pod \"d93351b1-967d-4f4d-9eb7-f41756ad3241\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") "
Sep 15 19:20:16.835055 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835013    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93351b1-967d-4f4d-9eb7-f41756ad3241-tls-certs\") pod \"d93351b1-967d-4f4d-9eb7-f41756ad3241\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") "
Sep 15 19:20:16.835222 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835084    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-home\") pod \"d93351b1-967d-4f4d-9eb7-f41756ad3241\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") "
Sep 15 19:20:16.835222 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835160    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-tmp-dir\") pod \"d93351b1-967d-4f4d-9eb7-f41756ad3241\" (UID: \"d93351b1-967d-4f4d-9eb7-f41756ad3241\") "
Sep 15 19:20:16.835332 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835214    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-model-cache" (OuterVolumeSpecName: "model-cache") pod "d93351b1-967d-4f4d-9eb7-f41756ad3241" (UID: "d93351b1-967d-4f4d-9eb7-f41756ad3241"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835451    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d93351b1-967d-4f4d-9eb7-f41756ad3241" (UID: "d93351b1-967d-4f4d-9eb7-f41756ad3241"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835473    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-home" (OuterVolumeSpecName: "home") pod "d93351b1-967d-4f4d-9eb7-f41756ad3241" (UID: "d93351b1-967d-4f4d-9eb7-f41756ad3241"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835597    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-tmp-dir\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835619    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55bgs\" (UniqueName: \"kubernetes.io/projected/9ab597c1-ab83-411c-91b1-285d384a64da-kube-api-access-55bgs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835634    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-model-cache\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835669    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab597c1-ab83-411c-91b1-285d384a64da-tls-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835682    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-dshm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835695    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-home\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.835754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.835710    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab597c1-ab83-411c-91b1-285d384a64da-model-cache\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.839280 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.839258    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-dshm" (OuterVolumeSpecName: "dshm") pod "d93351b1-967d-4f4d-9eb7-f41756ad3241" (UID: "d93351b1-967d-4f4d-9eb7-f41756ad3241"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.839385 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.839353    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93351b1-967d-4f4d-9eb7-f41756ad3241-kube-api-access-twwhf" (OuterVolumeSpecName: "kube-api-access-twwhf") pod "d93351b1-967d-4f4d-9eb7-f41756ad3241" (UID: "d93351b1-967d-4f4d-9eb7-f41756ad3241"). InnerVolumeSpecName "kube-api-access-twwhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:16.839385 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.839361    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d93351b1-967d-4f4d-9eb7-f41756ad3241-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d93351b1-967d-4f4d-9eb7-f41756ad3241" (UID: "d93351b1-967d-4f4d-9eb7-f41756ad3241"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:16.936894 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.936814    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twwhf\" (UniqueName: \"kubernetes.io/projected/d93351b1-967d-4f4d-9eb7-f41756ad3241-kube-api-access-twwhf\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.936894 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.936847    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93351b1-967d-4f4d-9eb7-f41756ad3241-dshm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.936894 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:16.936856    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93351b1-967d-4f4d-9eb7-f41756ad3241-tls-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:17.499994 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.499959    2556 generic.go:356] "Generic (PLEG): container finished" podID="d93351b1-967d-4f4d-9eb7-f41756ad3241" containerID="52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae" exitCode=0
Sep 15 19:20:17.500438 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.500001    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"
Sep 15 19:20:17.500438 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.500039    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz" event={"ID":"d93351b1-967d-4f4d-9eb7-f41756ad3241","Type":"ContainerDied","Data":"52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae"}
Sep 15 19:20:17.500438 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.500082    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz" event={"ID":"d93351b1-967d-4f4d-9eb7-f41756ad3241","Type":"ContainerDied","Data":"1ace36921e9c98942b105a3f95bb028d8ae3ac5f3825ad55d6b664329228bd7a"}
Sep 15 19:20:17.500438 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.500103    2556 scope.go:117] "RemoveContainer" containerID="52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae"
Sep 15 19:20:17.501633 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.501610    2556 generic.go:356] "Generic (PLEG): container finished" podID="9ab597c1-ab83-411c-91b1-285d384a64da" containerID="d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca" exitCode=0
Sep 15 19:20:17.501750 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.501671    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"
Sep 15 19:20:17.501850 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.501669    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65" event={"ID":"9ab597c1-ab83-411c-91b1-285d384a64da","Type":"ContainerDied","Data":"d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca"}
Sep 15 19:20:17.501985 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.501873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65" event={"ID":"9ab597c1-ab83-411c-91b1-285d384a64da","Type":"ContainerDied","Data":"c51f2c21c20fec9b53b9e30ef5ef84f79ea949460e98a75e0649818fc137e747"}
Sep 15 19:20:17.511500 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.511485    2556 scope.go:117] "RemoveContainer" containerID="52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae"
Sep 15 19:20:17.512128 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:20:17.511815    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae\": container with ID starting with 52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae not found: ID does not exist" containerID="52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae"
Sep 15 19:20:17.512128 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.512141    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae"} err="failed to get container status \"52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae\": rpc error: code = NotFound desc = could not find container \"52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae\": container with ID starting with 52dc2c2390e282571bba43a8ca81bd3917f31a75e9cd7ff4816791993284a5ae not found: ID does not exist"
Sep 15 19:20:17.512317 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.512163    2556 scope.go:117] "RemoveContainer" containerID="d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca"
Sep 15 19:20:17.521107 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.521087    2556 scope.go:117] "RemoveContainer" containerID="d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca"
Sep 15 19:20:17.521385 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:20:17.521364    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca\": container with ID starting with d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca not found: ID does not exist" containerID="d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca"
Sep 15 19:20:17.521442 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.521395    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca"} err="failed to get container status \"d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca\": rpc error: code = NotFound desc = could not find container \"d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca\": container with ID starting with d31a27da26d78d3cde74fb6bf070a282e989e978f6d030c6913627bc7cf43aca not found: ID does not exist"
Sep 15 19:20:17.526216 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.526180    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"]
Sep 15 19:20:17.530130 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.530110    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-2-w0-900c88-kserve-64995598cb-nxs65"]
Sep 15 19:20:17.538865 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.538844    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"]
Sep 15 19:20:17.544945 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.544925    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/embedding-w0-900c88-kserve-86b8df66b8-hx9pz"]
Sep 15 19:20:17.679150 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.679067    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab597c1-ab83-411c-91b1-285d384a64da" path="/var/lib/kubelet/pods/9ab597c1-ab83-411c-91b1-285d384a64da/volumes"
Sep 15 19:20:17.679512 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:17.679498    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d93351b1-967d-4f4d-9eb7-f41756ad3241" path="/var/lib/kubelet/pods/d93351b1-967d-4f4d-9eb7-f41756ad3241/volumes"
Sep 15 19:20:39.671795 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:39.671760    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:20:41.835000 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:41.834899    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:20:41.851859 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:41.843951    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:20:41.851859 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:41.844007    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:20:41.851859 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:41.848933    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:20:55.071991 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.071960    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"]
Sep 15 19:20:55.072825 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.072219    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64" podUID="bc0df392-dde3-421c-a312-b4ee50ab3ad6" containerName="payload-processing" containerID="cri-o://df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407" gracePeriod=30
Sep 15 19:20:55.079148 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.079109    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"]
Sep 15 19:20:55.079580 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.079545    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b" podUID="68dc6f53-9def-4871-99d2-cb691736750f" containerName="payload-pre-processing" containerID="cri-o://f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0" gracePeriod=30
Sep 15 19:20:55.369968 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.369944    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:20:55.373057 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.373037    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:20:55.509971 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.509941    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-744jb\" (UniqueName: \"kubernetes.io/projected/bc0df392-dde3-421c-a312-b4ee50ab3ad6-kube-api-access-744jb\") pod \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\" (UID: \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\") "
Sep 15 19:20:55.510137 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.509989    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dc6f53-9def-4871-99d2-cb691736750f-plugins-config-volume\") pod \"68dc6f53-9def-4871-99d2-cb691736750f\" (UID: \"68dc6f53-9def-4871-99d2-cb691736750f\") "
Sep 15 19:20:55.510137 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.510081    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhbbx\" (UniqueName: \"kubernetes.io/projected/68dc6f53-9def-4871-99d2-cb691736750f-kube-api-access-jhbbx\") pod \"68dc6f53-9def-4871-99d2-cb691736750f\" (UID: \"68dc6f53-9def-4871-99d2-cb691736750f\") "
Sep 15 19:20:55.510227 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.510150    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0df392-dde3-421c-a312-b4ee50ab3ad6-plugins-config-volume\") pod \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\" (UID: \"bc0df392-dde3-421c-a312-b4ee50ab3ad6\") "
Sep 15 19:20:55.511009 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.510740    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dc6f53-9def-4871-99d2-cb691736750f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68dc6f53-9def-4871-99d2-cb691736750f" (UID: "68dc6f53-9def-4871-99d2-cb691736750f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:55.511157 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.511086    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0df392-dde3-421c-a312-b4ee50ab3ad6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc0df392-dde3-421c-a312-b4ee50ab3ad6" (UID: "bc0df392-dde3-421c-a312-b4ee50ab3ad6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:55.511844 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.511237    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dc6f53-9def-4871-99d2-cb691736750f-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:55.511844 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.511262    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc0df392-dde3-421c-a312-b4ee50ab3ad6-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:55.513029 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.512994    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0df392-dde3-421c-a312-b4ee50ab3ad6-kube-api-access-744jb" (OuterVolumeSpecName: "kube-api-access-744jb") pod "bc0df392-dde3-421c-a312-b4ee50ab3ad6" (UID: "bc0df392-dde3-421c-a312-b4ee50ab3ad6"). InnerVolumeSpecName "kube-api-access-744jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:55.520986 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.520959    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68dc6f53-9def-4871-99d2-cb691736750f-kube-api-access-jhbbx" (OuterVolumeSpecName: "kube-api-access-jhbbx") pod "68dc6f53-9def-4871-99d2-cb691736750f" (UID: "68dc6f53-9def-4871-99d2-cb691736750f"). InnerVolumeSpecName "kube-api-access-jhbbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:55.612039 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.611963    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbbx\" (UniqueName: \"kubernetes.io/projected/68dc6f53-9def-4871-99d2-cb691736750f-kube-api-access-jhbbx\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:55.612039 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.611995    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-744jb\" (UniqueName: \"kubernetes.io/projected/bc0df392-dde3-421c-a312-b4ee50ab3ad6-kube-api-access-744jb\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:20:55.711256 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.711230    2556 generic.go:356] "Generic (PLEG): container finished" podID="bc0df392-dde3-421c-a312-b4ee50ab3ad6" containerID="df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407" exitCode=0
Sep 15 19:20:55.711367 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.711282    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"
Sep 15 19:20:55.711367 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.711315    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64" event={"ID":"bc0df392-dde3-421c-a312-b4ee50ab3ad6","Type":"ContainerDied","Data":"df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407"}
Sep 15 19:20:55.711367 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.711363    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64" event={"ID":"bc0df392-dde3-421c-a312-b4ee50ab3ad6","Type":"ContainerDied","Data":"9746c549b76ac47057a5b8fffb9a30284208eeec5da07af21acebd89b7fd3d64"}
Sep 15 19:20:55.711496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.711386    2556 scope.go:117] "RemoveContainer" containerID="df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407"
Sep 15 19:20:55.712878 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.712852    2556 generic.go:356] "Generic (PLEG): container finished" podID="68dc6f53-9def-4871-99d2-cb691736750f" containerID="f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0" exitCode=0
Sep 15 19:20:55.712992 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.712890    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"
Sep 15 19:20:55.712992 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.712896    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b" event={"ID":"68dc6f53-9def-4871-99d2-cb691736750f","Type":"ContainerDied","Data":"f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0"}
Sep 15 19:20:55.712992 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.712938    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b" event={"ID":"68dc6f53-9def-4871-99d2-cb691736750f","Type":"ContainerDied","Data":"797632658b1bd10536741d6943a96fbc3ffd5bc06598deb34fff034ee0ab7135"}
Sep 15 19:20:55.720869 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.720853    2556 scope.go:117] "RemoveContainer" containerID="df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407"
Sep 15 19:20:55.721139 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:20:55.721120    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407\": container with ID starting with df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407 not found: ID does not exist" containerID="df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407"
Sep 15 19:20:55.721198 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.721145    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407"} err="failed to get container status \"df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407\": rpc error: code = NotFound desc = could not find container \"df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407\": container with ID starting with df70c8667cffb5aca2ff16f7047cacd2bdc22793c7f5bccd73d0cc785c10e407 not found: ID does not exist"
Sep 15 19:20:55.721198 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.721161    2556 scope.go:117] "RemoveContainer" containerID="f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0"
Sep 15 19:20:55.728134 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.728108    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"]
Sep 15 19:20:55.729860 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.729838    2556 scope.go:117] "RemoveContainer" containerID="f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0"
Sep 15 19:20:55.730155 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:20:55.730137    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0\": container with ID starting with f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0 not found: ID does not exist" containerID="f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0"
Sep 15 19:20:55.730218 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.730162    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0"} err="failed to get container status \"f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0\": rpc error: code = NotFound desc = could not find container \"f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0\": container with ID starting with f2b5fc9b87e5fdd48822f66418a49e0285a4a6ec5e47ad24c8c0e8b4b87729f0 not found: ID does not exist"
Sep 15 19:20:55.731529 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.731511    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-759580-587878fcd6-24f7b"]
Sep 15 19:20:55.741226 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.741204    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"]
Sep 15 19:20:55.745203 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:55.745183    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-759580-74684d8d79-fdx64"]
Sep 15 19:20:57.678881 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:57.678844    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68dc6f53-9def-4871-99d2-cb691736750f" path="/var/lib/kubelet/pods/68dc6f53-9def-4871-99d2-cb691736750f/volumes"
Sep 15 19:20:57.679383 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:20:57.679364    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc0df392-dde3-421c-a312-b4ee50ab3ad6" path="/var/lib/kubelet/pods/bc0df392-dde3-421c-a312-b4ee50ab3ad6/volumes"
Sep 15 19:21:09.678131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:09.678100    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:21:35.164837 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:35.164790    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"]
Sep 15 19:21:41.914314 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:41.914282    2556 scope.go:117] "RemoveContainer" containerID="3fa8b31a90e89b132f3dade49323426a936636ca46cc44b60924bc5a326754a9"
Sep 15 19:21:45.672524 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:45.672493    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:21:50.186425 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.186364    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2" podUID="aba7ed5c-4512-4ab5-be20-80608659f886" containerName="main" containerID="cri-o://bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5" gracePeriod=45
Sep 15 19:21:50.442712 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.442655    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:21:50.538869 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.538839    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-tmp-dir\") pod \"aba7ed5c-4512-4ab5-be20-80608659f886\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") "
Sep 15 19:21:50.538869 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.538871    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-dshm\") pod \"aba7ed5c-4512-4ab5-be20-80608659f886\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") "
Sep 15 19:21:50.539076 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.538957    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba7ed5c-4512-4ab5-be20-80608659f886-tls-certs\") pod \"aba7ed5c-4512-4ab5-be20-80608659f886\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") "
Sep 15 19:21:50.539076 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.539021    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-model-cache\") pod \"aba7ed5c-4512-4ab5-be20-80608659f886\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") "
Sep 15 19:21:50.539076 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.539053    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lthf9\" (UniqueName: \"kubernetes.io/projected/aba7ed5c-4512-4ab5-be20-80608659f886-kube-api-access-lthf9\") pod \"aba7ed5c-4512-4ab5-be20-80608659f886\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") "
Sep 15 19:21:50.539236 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.539089    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-home\") pod \"aba7ed5c-4512-4ab5-be20-80608659f886\" (UID: \"aba7ed5c-4512-4ab5-be20-80608659f886\") "
Sep 15 19:21:50.539236 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.539089    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aba7ed5c-4512-4ab5-be20-80608659f886" (UID: "aba7ed5c-4512-4ab5-be20-80608659f886"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:50.539345 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.539256    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-model-cache" (OuterVolumeSpecName: "model-cache") pod "aba7ed5c-4512-4ab5-be20-80608659f886" (UID: "aba7ed5c-4512-4ab5-be20-80608659f886"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:50.539457 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.539410    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-home" (OuterVolumeSpecName: "home") pod "aba7ed5c-4512-4ab5-be20-80608659f886" (UID: "aba7ed5c-4512-4ab5-be20-80608659f886"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:50.539594 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.539423    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-model-cache\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:50.539594 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.539487    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-tmp-dir\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:50.541270 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.541242    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba7ed5c-4512-4ab5-be20-80608659f886-kube-api-access-lthf9" (OuterVolumeSpecName: "kube-api-access-lthf9") pod "aba7ed5c-4512-4ab5-be20-80608659f886" (UID: "aba7ed5c-4512-4ab5-be20-80608659f886"). InnerVolumeSpecName "kube-api-access-lthf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:50.541270 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.541257    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-dshm" (OuterVolumeSpecName: "dshm") pod "aba7ed5c-4512-4ab5-be20-80608659f886" (UID: "aba7ed5c-4512-4ab5-be20-80608659f886"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:50.541413 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.541250    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba7ed5c-4512-4ab5-be20-80608659f886-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aba7ed5c-4512-4ab5-be20-80608659f886" (UID: "aba7ed5c-4512-4ab5-be20-80608659f886"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:50.640211 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.640181    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-dshm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:50.640211 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.640204    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba7ed5c-4512-4ab5-be20-80608659f886-tls-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:50.640211 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.640214    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lthf9\" (UniqueName: \"kubernetes.io/projected/aba7ed5c-4512-4ab5-be20-80608659f886-kube-api-access-lthf9\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:50.640401 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:50.640224    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba7ed5c-4512-4ab5-be20-80608659f886-home\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:51.009786 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.009754    2556 generic.go:356] "Generic (PLEG): container finished" podID="aba7ed5c-4512-4ab5-be20-80608659f886" containerID="bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5" exitCode=0
Sep 15 19:21:51.009926 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.009798    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"
Sep 15 19:21:51.009926 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.009833    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2" event={"ID":"aba7ed5c-4512-4ab5-be20-80608659f886","Type":"ContainerDied","Data":"bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5"}
Sep 15 19:21:51.009926 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.009875    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2" event={"ID":"aba7ed5c-4512-4ab5-be20-80608659f886","Type":"ContainerDied","Data":"124d486dc47e38a8aa1756937fd072e4711fc332e56d9fa19bcf60c62a1d3dbc"}
Sep 15 19:21:51.009926 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.009895    2556 scope.go:117] "RemoveContainer" containerID="bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5"
Sep 15 19:21:51.019868 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.019851    2556 scope.go:117] "RemoveContainer" containerID="bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5"
Sep 15 19:21:51.020138 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:21:51.020120    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5\": container with ID starting with bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5 not found: ID does not exist" containerID="bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5"
Sep 15 19:21:51.020184 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.020148    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5"} err="failed to get container status \"bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5\": rpc error: code = NotFound desc = could not find container \"bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5\": container with ID starting with bddb8e10c743e3543d5cb894a5f3553b76868f5562ca77a9fbd28200eccb0fc5 not found: ID does not exist"
Sep 15 19:21:51.031815 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.031793    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"]
Sep 15 19:21:51.035226 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.035208    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/auth-test-model-415f87-kserve-98dbc7d79-sdsm2"]
Sep 15 19:21:51.678979 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:51.678945    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba7ed5c-4512-4ab5-be20-80608659f886" path="/var/lib/kubelet/pods/aba7ed5c-4512-4ab5-be20-80608659f886/volumes"
Sep 15 19:21:52.968012 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:52.967982    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"]
Sep 15 19:21:52.968425 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:52.968207    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4" podUID="6df2bd20-a2e1-4ca2-9d02-0934804bab25" containerName="maas-api" containerID="cri-o://2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb" gracePeriod=30
Sep 15 19:21:53.010513 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.010477    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"]
Sep 15 19:21:53.011026 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.010979    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g" podUID="6aab0344-ae04-4211-b2cf-6366e000d19d" containerName="payload-processing" containerID="cri-o://cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430" gracePeriod=30
Sep 15 19:21:53.255544 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.255515    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:21:53.294887 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.294860    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:21:53.365265 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.365233    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aab0344-ae04-4211-b2cf-6366e000d19d-plugins-config-volume\") pod \"6aab0344-ae04-4211-b2cf-6366e000d19d\" (UID: \"6aab0344-ae04-4211-b2cf-6366e000d19d\") "
Sep 15 19:21:53.365459 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.365363    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls\") pod \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") "
Sep 15 19:21:53.365459 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.365413    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-metrics-certs\") pod \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") "
Sep 15 19:21:53.365459 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.365451    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhkr5\" (UniqueName: \"kubernetes.io/projected/6df2bd20-a2e1-4ca2-9d02-0934804bab25-kube-api-access-hhkr5\") pod \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") "
Sep 15 19:21:53.365630 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.365489    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrng9\" (UniqueName: \"kubernetes.io/projected/6aab0344-ae04-4211-b2cf-6366e000d19d-kube-api-access-lrng9\") pod \"6aab0344-ae04-4211-b2cf-6366e000d19d\" (UID: \"6aab0344-ae04-4211-b2cf-6366e000d19d\") "
Sep 15 19:21:53.365630 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.365517    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df2bd20-a2e1-4ca2-9d02-0934804bab25-openshift-service-ca\") pod \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\" (UID: \"6df2bd20-a2e1-4ca2-9d02-0934804bab25\") "
Sep 15 19:21:53.365778 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.365625    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aab0344-ae04-4211-b2cf-6366e000d19d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6aab0344-ae04-4211-b2cf-6366e000d19d" (UID: "6aab0344-ae04-4211-b2cf-6366e000d19d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:53.365929 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.365900    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aab0344-ae04-4211-b2cf-6366e000d19d-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:53.366152 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.366114    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df2bd20-a2e1-4ca2-9d02-0934804bab25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6df2bd20-a2e1-4ca2-9d02-0934804bab25" (UID: "6df2bd20-a2e1-4ca2-9d02-0934804bab25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:53.367720 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.367690    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df2bd20-a2e1-4ca2-9d02-0934804bab25-kube-api-access-hhkr5" (OuterVolumeSpecName: "kube-api-access-hhkr5") pod "6df2bd20-a2e1-4ca2-9d02-0934804bab25" (UID: "6df2bd20-a2e1-4ca2-9d02-0934804bab25"). InnerVolumeSpecName "kube-api-access-hhkr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:53.367840 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.367811    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aab0344-ae04-4211-b2cf-6366e000d19d-kube-api-access-lrng9" (OuterVolumeSpecName: "kube-api-access-lrng9") pod "6aab0344-ae04-4211-b2cf-6366e000d19d" (UID: "6aab0344-ae04-4211-b2cf-6366e000d19d"). InnerVolumeSpecName "kube-api-access-lrng9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:53.368011 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.367978    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6df2bd20-a2e1-4ca2-9d02-0934804bab25" (UID: "6df2bd20-a2e1-4ca2-9d02-0934804bab25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:53.368108 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.368069    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6df2bd20-a2e1-4ca2-9d02-0934804bab25" (UID: "6df2bd20-a2e1-4ca2-9d02-0934804bab25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:53.466863 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.466832    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-maas-api-tls\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:53.466863 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.466857    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df2bd20-a2e1-4ca2-9d02-0934804bab25-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:53.466863 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.466867    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhkr5\" (UniqueName: \"kubernetes.io/projected/6df2bd20-a2e1-4ca2-9d02-0934804bab25-kube-api-access-hhkr5\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:53.467076 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.466876    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrng9\" (UniqueName: \"kubernetes.io/projected/6aab0344-ae04-4211-b2cf-6366e000d19d-kube-api-access-lrng9\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:53.467076 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:53.466885    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df2bd20-a2e1-4ca2-9d02-0934804bab25-openshift-service-ca\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:21:54.027796 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.027764    2556 generic.go:356] "Generic (PLEG): container finished" podID="6df2bd20-a2e1-4ca2-9d02-0934804bab25" containerID="2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb" exitCode=0
Sep 15 19:21:54.028276 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.027812    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"
Sep 15 19:21:54.028276 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.027845    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4" event={"ID":"6df2bd20-a2e1-4ca2-9d02-0934804bab25","Type":"ContainerDied","Data":"2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb"}
Sep 15 19:21:54.028276 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.027897    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4" event={"ID":"6df2bd20-a2e1-4ca2-9d02-0934804bab25","Type":"ContainerDied","Data":"69f04724cf60f3d095a0c26b31478cdc66495d6533b063fd6f240c3473918a2c"}
Sep 15 19:21:54.028276 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.027927    2556 scope.go:117] "RemoveContainer" containerID="2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb"
Sep 15 19:21:54.030233 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.030207    2556 generic.go:356] "Generic (PLEG): container finished" podID="6aab0344-ae04-4211-b2cf-6366e000d19d" containerID="cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430" exitCode=0
Sep 15 19:21:54.030353 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.030246    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g" event={"ID":"6aab0344-ae04-4211-b2cf-6366e000d19d","Type":"ContainerDied","Data":"cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430"}
Sep 15 19:21:54.030353 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.030271    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"
Sep 15 19:21:54.030353 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.030281    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g" event={"ID":"6aab0344-ae04-4211-b2cf-6366e000d19d","Type":"ContainerDied","Data":"5a6c4bf8f5fc2bb3841392fe4860c49758ad932840587d9c2f2b4735e8b604b6"}
Sep 15 19:21:54.038162 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.038143    2556 scope.go:117] "RemoveContainer" containerID="2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb"
Sep 15 19:21:54.038445 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:21:54.038417    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb\": container with ID starting with 2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb not found: ID does not exist" containerID="2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb"
Sep 15 19:21:54.038501 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.038453    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb"} err="failed to get container status \"2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb\": rpc error: code = NotFound desc = could not find container \"2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb\": container with ID starting with 2505f1ba8023eacbef5e0e30dd09c22ad22429e2eb5371ea9ca4d256b0db92eb not found: ID does not exist"
Sep 15 19:21:54.038501 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.038470    2556 scope.go:117] "RemoveContainer" containerID="cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430"
Sep 15 19:21:54.044618 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.044595    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"]
Sep 15 19:21:54.047818 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.047796    2556 scope.go:117] "RemoveContainer" containerID="cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430"
Sep 15 19:21:54.048167 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:21:54.048143    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430\": container with ID starting with cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430 not found: ID does not exist" containerID="cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430"
Sep 15 19:21:54.048269 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.048176    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430"} err="failed to get container status \"cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430\": rpc error: code = NotFound desc = could not find container \"cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430\": container with ID starting with cc810562229dfc65e2d8fb26019735eb7874c3c3be8a0a428947dce90ca39430 not found: ID does not exist"
Sep 15 19:21:54.048269 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.048219    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1b0e3f-57557b7794-rfng4"]
Sep 15 19:21:54.059820 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.059796    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"]
Sep 15 19:21:54.066549 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:54.066499    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1b0e3f-87f769c5-xrc6g"]
Sep 15 19:21:55.678893 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:55.678863    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aab0344-ae04-4211-b2cf-6366e000d19d" path="/var/lib/kubelet/pods/6aab0344-ae04-4211-b2cf-6366e000d19d/volumes"
Sep 15 19:21:55.679260 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:21:55.679231    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df2bd20-a2e1-4ca2-9d02-0934804bab25" path="/var/lib/kubelet/pods/6df2bd20-a2e1-4ca2-9d02-0934804bab25/volumes"
Sep 15 19:22:11.674515 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:11.674420    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:22:43.059139 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059101    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9002-595dcf"]
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059509    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aab0344-ae04-4211-b2cf-6366e000d19d" containerName="payload-processing"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059520    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aab0344-ae04-4211-b2cf-6366e000d19d" containerName="payload-processing"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059530    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba7ed5c-4512-4ab5-be20-80608659f886" containerName="main"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059535    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba7ed5c-4512-4ab5-be20-80608659f886" containerName="main"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059548    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0df392-dde3-421c-a312-b4ee50ab3ad6" containerName="payload-processing"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059554    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0df392-dde3-421c-a312-b4ee50ab3ad6" containerName="payload-processing"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059560    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68dc6f53-9def-4871-99d2-cb691736750f" containerName="payload-pre-processing"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059565    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="68dc6f53-9def-4871-99d2-cb691736750f" containerName="payload-pre-processing"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059573    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab597c1-ab83-411c-91b1-285d384a64da" containerName="main"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059578    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab597c1-ab83-411c-91b1-285d384a64da" containerName="main"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059585    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df2bd20-a2e1-4ca2-9d02-0934804bab25" containerName="maas-api"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059590    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df2bd20-a2e1-4ca2-9d02-0934804bab25" containerName="maas-api"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059603    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93351b1-967d-4f4d-9eb7-f41756ad3241" containerName="main"
Sep 15 19:22:43.059601 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059609    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93351b1-967d-4f4d-9eb7-f41756ad3241" containerName="main"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059615    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fcaeca-caa2-46f1-8b42-922e878ed390" containerName="maas-api"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059620    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fcaeca-caa2-46f1-8b42-922e878ed390" containerName="maas-api"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059700    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df2bd20-a2e1-4ca2-9d02-0934804bab25" containerName="maas-api"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059708    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="68dc6f53-9def-4871-99d2-cb691736750f" containerName="payload-pre-processing"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059713    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aab0344-ae04-4211-b2cf-6366e000d19d" containerName="payload-processing"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059720    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc0df392-dde3-421c-a312-b4ee50ab3ad6" containerName="payload-processing"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059725    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93351b1-967d-4f4d-9eb7-f41756ad3241" containerName="main"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059730    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab597c1-ab83-411c-91b1-285d384a64da" containerName="main"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059737    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba7ed5c-4512-4ab5-be20-80608659f886" containerName="main"
Sep 15 19:22:43.060066 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.059743    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fcaeca-caa2-46f1-8b42-922e878ed390" containerName="maas-api"
Sep 15 19:22:43.064049 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.064031    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-595dcf"
Sep 15 19:22:43.066403 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.066378    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgkcn\""
Sep 15 19:22:43.069497 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.069471    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9002-595dcf"]
Sep 15 19:22:43.136019 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.135987    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngs9v\" (UniqueName: \"kubernetes.io/projected/0d5e9386-4250-4790-afde-b56100f73a7f-kube-api-access-ngs9v\") pod \"test-curl-9002-595dcf\" (UID: \"0d5e9386-4250-4790-afde-b56100f73a7f\") " pod="opendatahub/test-curl-9002-595dcf"
Sep 15 19:22:43.236948 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.236917    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngs9v\" (UniqueName: \"kubernetes.io/projected/0d5e9386-4250-4790-afde-b56100f73a7f-kube-api-access-ngs9v\") pod \"test-curl-9002-595dcf\" (UID: \"0d5e9386-4250-4790-afde-b56100f73a7f\") " pod="opendatahub/test-curl-9002-595dcf"
Sep 15 19:22:43.244815 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.244793    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngs9v\" (UniqueName: \"kubernetes.io/projected/0d5e9386-4250-4790-afde-b56100f73a7f-kube-api-access-ngs9v\") pod \"test-curl-9002-595dcf\" (UID: \"0d5e9386-4250-4790-afde-b56100f73a7f\") " pod="opendatahub/test-curl-9002-595dcf"
Sep 15 19:22:43.375323 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.375296    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-595dcf"
Sep 15 19:22:43.642412 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:43.642390    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9002-595dcf"]
Sep 15 19:22:43.644427 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:22:43.644394    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d5e9386_4250_4790_afde_b56100f73a7f.slice/crio-362bd164148c225d045af89b2eac4c5e1d7b412888f9463f24281abdb1496f84 WatchSource:0}: Error finding container 362bd164148c225d045af89b2eac4c5e1d7b412888f9463f24281abdb1496f84: Status 404 returned error can't find the container with id 362bd164148c225d045af89b2eac4c5e1d7b412888f9463f24281abdb1496f84
Sep 15 19:22:44.297939 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.297901    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9002-595dcf" event={"ID":"0d5e9386-4250-4790-afde-b56100f73a7f","Type":"ContainerStarted","Data":"56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a"}
Sep 15 19:22:44.298311 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.297947    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9002-595dcf" event={"ID":"0d5e9386-4250-4790-afde-b56100f73a7f","Type":"ContainerStarted","Data":"362bd164148c225d045af89b2eac4c5e1d7b412888f9463f24281abdb1496f84"}
Sep 15 19:22:44.313158 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.313104    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9002-595dcf" podStartSLOduration=1.313086458 podStartE2EDuration="1.313086458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:22:44.310859253 +0000 UTC m=+1923.166712531" watchObservedRunningTime="2026-09-15 19:22:44.313086458 +0000 UTC m=+1923.168939770"
Sep 15 19:22:44.879243 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.879206    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9002-595dcf"]
Sep 15 19:22:44.883252 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.883213    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9002-595dcf"]
Sep 15 19:22:44.991096 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.991016    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9002-153a64"]
Sep 15 19:22:44.991433 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.991421    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5e9386-4250-4790-afde-b56100f73a7f" containerName="test-curl-9002-595dcf"
Sep 15 19:22:44.991477 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.991434    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5e9386-4250-4790-afde-b56100f73a7f" containerName="test-curl-9002-595dcf"
Sep 15 19:22:44.991521 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.991512    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5e9386-4250-4790-afde-b56100f73a7f" containerName="test-curl-9002-595dcf"
Sep 15 19:22:44.994459 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.994444    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:44.998780 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:44.998757    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9002-153a64"]
Sep 15 19:22:45.156178 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:45.156141    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mvl\" (UniqueName: \"kubernetes.io/projected/862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c-kube-api-access-c9mvl\") pod \"test-curl-9002-153a64\" (UID: \"862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c\") " pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:45.257467 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:45.257383    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mvl\" (UniqueName: \"kubernetes.io/projected/862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c-kube-api-access-c9mvl\") pod \"test-curl-9002-153a64\" (UID: \"862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c\") " pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:45.265321 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:45.265285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mvl\" (UniqueName: \"kubernetes.io/projected/862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c-kube-api-access-c9mvl\") pod \"test-curl-9002-153a64\" (UID: \"862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c\") " pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:45.302836 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:22:45.302812    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9002-595dcf" container="test-curl-9002-595dcf"
Sep 15 19:22:45.304751 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:45.304736    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:45.571836 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:45.571762    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9002-153a64"]
Sep 15 19:22:45.575283 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:22:45.575254    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod862289d3_8b3c_42ae_a9d5_9c1c3c78eb3c.slice/crio-ac0c50bffbbccfe99eca7800f83058460bfa1693d34a77301ed0a8789e2c6c17 WatchSource:0}: Error finding container ac0c50bffbbccfe99eca7800f83058460bfa1693d34a77301ed0a8789e2c6c17: Status 404 returned error can't find the container with id ac0c50bffbbccfe99eca7800f83058460bfa1693d34a77301ed0a8789e2c6c17
Sep 15 19:22:46.309610 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:46.309573    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9002-153a64" event={"ID":"862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c","Type":"ContainerStarted","Data":"a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf"}
Sep 15 19:22:46.309610 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:46.309616    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9002-153a64" event={"ID":"862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c","Type":"ContainerStarted","Data":"ac0c50bffbbccfe99eca7800f83058460bfa1693d34a77301ed0a8789e2c6c17"}
Sep 15 19:22:46.310088 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:46.309592    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9002-595dcf" podUID="0d5e9386-4250-4790-afde-b56100f73a7f" containerName="test-curl-9002-595dcf" containerID="cri-o://56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a" gracePeriod=2
Sep 15 19:22:46.323318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:46.323273    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9002-153a64" podStartSLOduration=2.3232569180000002 podStartE2EDuration="2.323256918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:22:46.322788186 +0000 UTC m=+1925.178641454" watchObservedRunningTime="2026-09-15 19:22:46.323256918 +0000 UTC m=+1925.179110186"
Sep 15 19:22:46.871286 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:46.871211    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9002-153a64"]
Sep 15 19:22:46.873692 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:46.873661    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9002-153a64"]
Sep 15 19:22:47.316591 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:22:47.316552    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9002-153a64" container="test-curl-9002-153a64"
Sep 15 19:22:48.324362 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:48.324325    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9002-153a64" podUID="862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" containerName="test-curl-9002-153a64" containerID="cri-o://a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf" gracePeriod=2
Sep 15 19:22:48.497394 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:48.497368    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-595dcf"
Sep 15 19:22:48.590472 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:48.590387    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngs9v\" (UniqueName: \"kubernetes.io/projected/0d5e9386-4250-4790-afde-b56100f73a7f-kube-api-access-ngs9v\") pod \"0d5e9386-4250-4790-afde-b56100f73a7f\" (UID: \"0d5e9386-4250-4790-afde-b56100f73a7f\") "
Sep 15 19:22:48.592592 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:48.592562    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5e9386-4250-4790-afde-b56100f73a7f-kube-api-access-ngs9v" (OuterVolumeSpecName: "kube-api-access-ngs9v") pod "0d5e9386-4250-4790-afde-b56100f73a7f" (UID: "0d5e9386-4250-4790-afde-b56100f73a7f"). InnerVolumeSpecName "kube-api-access-ngs9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:48.691407 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:48.691376    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngs9v\" (UniqueName: \"kubernetes.io/projected/0d5e9386-4250-4790-afde-b56100f73a7f-kube-api-access-ngs9v\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:22:49.330398 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:49.330362    2556 generic.go:356] "Generic (PLEG): container finished" podID="0d5e9386-4250-4790-afde-b56100f73a7f" containerID="56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a" exitCode=137
Sep 15 19:22:49.330852 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:49.330411    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-595dcf"
Sep 15 19:22:49.330852 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:49.330456    2556 scope.go:117] "RemoveContainer" containerID="56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a"
Sep 15 19:22:49.339325 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:49.339308    2556 scope.go:117] "RemoveContainer" containerID="56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a"
Sep 15 19:22:49.339572 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:22:49.339550    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a\": container with ID starting with 56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a not found: ID does not exist" containerID="56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a"
Sep 15 19:22:49.339623 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:49.339584    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a"} err="failed to get container status \"56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a\": rpc error: code = NotFound desc = could not find container \"56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a\": container with ID starting with 56729c891b2289300897ed455a6a69e12dcf9d78ae51437e84b2f7320554a51a not found: ID does not exist"
Sep 15 19:22:49.340621 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:22:49.340600    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9002-595dcf\" is forbidden: User \"system:node:ip-10-0-140-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-183.ec2.internal' and this object" podUID="0d5e9386-4250-4790-afde-b56100f73a7f" pod="opendatahub/test-curl-9002-595dcf"
Sep 15 19:22:49.677935 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:49.677899    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5e9386-4250-4790-afde-b56100f73a7f" path="/var/lib/kubelet/pods/0d5e9386-4250-4790-afde-b56100f73a7f/volumes"
Sep 15 19:22:50.511861 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:50.511837    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:50.609839 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:50.609757    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9mvl\" (UniqueName: \"kubernetes.io/projected/862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c-kube-api-access-c9mvl\") pod \"862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c\" (UID: \"862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c\") "
Sep 15 19:22:50.611850 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:50.611829    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c-kube-api-access-c9mvl" (OuterVolumeSpecName: "kube-api-access-c9mvl") pod "862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" (UID: "862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c"). InnerVolumeSpecName "kube-api-access-c9mvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:50.710845 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:50.710812    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mvl\" (UniqueName: \"kubernetes.io/projected/862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c-kube-api-access-c9mvl\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:22:51.343527 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:51.343486    2556 generic.go:356] "Generic (PLEG): container finished" podID="862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" containerID="a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf" exitCode=137
Sep 15 19:22:51.343527 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:51.343527    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:51.343778 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:51.343580    2556 scope.go:117] "RemoveContainer" containerID="a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf"
Sep 15 19:22:51.359325 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:22:51.359294    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9002-153a64\" is forbidden: User \"system:node:ip-10-0-140-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-183.ec2.internal' and this object" podUID="862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:51.359687 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:51.359671    2556 scope.go:117] "RemoveContainer" containerID="a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf"
Sep 15 19:22:51.359936 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:22:51.359920    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf\": container with ID starting with a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf not found: ID does not exist" containerID="a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf"
Sep 15 19:22:51.359975 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:51.359943    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf"} err="failed to get container status \"a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf\": rpc error: code = NotFound desc = could not find container \"a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf\": container with ID starting with a61004fdd5e13a704f2e1cbb1838ca66c50a579ede54b05d4d3324e1ad187bdf not found: ID does not exist"
Sep 15 19:22:51.676079 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:22:51.676033    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9002-153a64\" is forbidden: User \"system:node:ip-10-0-140-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-183.ec2.internal' and this object" podUID="862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" pod="opendatahub/test-curl-9002-153a64"
Sep 15 19:22:51.678251 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:51.678227    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" path="/var/lib/kubelet/pods/862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c/volumes"
Sep 15 19:22:53.179609 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.179573    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"]
Sep 15 19:22:53.180030 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.180015    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" containerName="test-curl-9002-153a64"
Sep 15 19:22:53.180030 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.180030    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" containerName="test-curl-9002-153a64"
Sep 15 19:22:53.180118 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.180109    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="862289d3-8b3c-42ae-a9d5-9c1c3c78eb3c" containerName="test-curl-9002-153a64"
Sep 15 19:22:53.184187 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.184169    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.186663 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.186626    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"openshift-service-ca.crt\""
Sep 15 19:22:53.186785 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.186715    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"kube-root-ca.crt\""
Sep 15 19:22:53.187586 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.187567    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"default-dockercfg-hc5bs\""
Sep 15 19:22:53.187681 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.187611    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"rate-test-model-c6d10a-kserve-self-signed-certs\""
Sep 15 19:22:53.191885 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.191588    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"]
Sep 15 19:22:53.333496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.333464    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xzz\" (UniqueName: \"kubernetes.io/projected/56813081-07ff-49d3-af15-f769dec56750-kube-api-access-78xzz\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.333680 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.333513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56813081-07ff-49d3-af15-f769dec56750-tls-certs\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.333680 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.333532    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-dshm\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.333680 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.333586    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-model-cache\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.333797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.333691    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-home\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.333797 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.333743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-tmp-dir\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434445 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434369    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-tmp-dir\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434445 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434415    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xzz\" (UniqueName: \"kubernetes.io/projected/56813081-07ff-49d3-af15-f769dec56750-kube-api-access-78xzz\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434622 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434446    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56813081-07ff-49d3-af15-f769dec56750-tls-certs\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434622 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434462    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-dshm\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434622 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434491    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-model-cache\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434622 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434544    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-home\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434858 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434826    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-tmp-dir\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434976 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-model-cache\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.434976 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.434936    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-home\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.436806 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.436781    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-dshm\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.436977 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.436959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56813081-07ff-49d3-af15-f769dec56750-tls-certs\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.441610 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.441594    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xzz\" (UniqueName: \"kubernetes.io/projected/56813081-07ff-49d3-af15-f769dec56750-kube-api-access-78xzz\") pod \"rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.495332 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.495295    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:22:53.765790 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:53.765764    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"]
Sep 15 19:22:53.767573 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:22:53.767537    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56813081_07ff_49d3_af15_f769dec56750.slice/crio-1f14cce8444e309ea865f88beb0d2ccc87241232fc39eece4445665c0c284076 WatchSource:0}: Error finding container 1f14cce8444e309ea865f88beb0d2ccc87241232fc39eece4445665c0c284076: Status 404 returned error can't find the container with id 1f14cce8444e309ea865f88beb0d2ccc87241232fc39eece4445665c0c284076
Sep 15 19:22:54.362175 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:54.362135    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq" event={"ID":"56813081-07ff-49d3-af15-f769dec56750","Type":"ContainerStarted","Data":"ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705"}
Sep 15 19:22:54.362175 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:54.362181    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq" event={"ID":"56813081-07ff-49d3-af15-f769dec56750","Type":"ContainerStarted","Data":"1f14cce8444e309ea865f88beb0d2ccc87241232fc39eece4445665c0c284076"}
Sep 15 19:22:54.378568 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:22:54.378516    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq" podStartSLOduration=1.3784980039999999 podStartE2EDuration="1.378498004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:22:54.3767914 +0000 UTC m=+1933.232644668" watchObservedRunningTime="2026-09-15 19:22:54.378498004 +0000 UTC m=+1933.234351273"
Sep 15 19:23:03.496146 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:03.496111    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:23:03.496498 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:03.496164    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:23:03.508922 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:03.508895    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:23:04.426086 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:04.426058    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:23:12.672498 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:12.672467    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:23:33.672540 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:33.672505    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:23:47.812265 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:47.812232    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"]
Sep 15 19:23:55.439483 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.439437    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"]
Sep 15 19:23:55.443472 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.443448    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:55.445958 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.445938    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f61404-dockercfg-jrhqn\""
Sep 15 19:23:55.446068 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.445962    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f61404\""
Sep 15 19:23:55.462451 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.462399    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"]
Sep 15 19:23:55.526136 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.526093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dcl\" (UniqueName: \"kubernetes.io/projected/d54436b5-9066-4656-97c5-04fb36340a09-kube-api-access-88dcl\") pod \"payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2\" (UID: \"d54436b5-9066-4656-97c5-04fb36340a09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:55.526303 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.526205    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54436b5-9066-4656-97c5-04fb36340a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2\" (UID: \"d54436b5-9066-4656-97c5-04fb36340a09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:55.627414 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.627376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dcl\" (UniqueName: \"kubernetes.io/projected/d54436b5-9066-4656-97c5-04fb36340a09-kube-api-access-88dcl\") pod \"payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2\" (UID: \"d54436b5-9066-4656-97c5-04fb36340a09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:55.627627 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.627440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54436b5-9066-4656-97c5-04fb36340a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2\" (UID: \"d54436b5-9066-4656-97c5-04fb36340a09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:55.628155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.628134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54436b5-9066-4656-97c5-04fb36340a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2\" (UID: \"d54436b5-9066-4656-97c5-04fb36340a09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:55.635596 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.635574    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dcl\" (UniqueName: \"kubernetes.io/projected/d54436b5-9066-4656-97c5-04fb36340a09-kube-api-access-88dcl\") pod \"payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2\" (UID: \"d54436b5-9066-4656-97c5-04fb36340a09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:55.756089 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:55.756008    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:56.035106 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:56.035078    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"]
Sep 15 19:23:56.037182 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:23:56.037154    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54436b5_9066_4656_97c5_04fb36340a09.slice/crio-a7607af822c42795b92e235e6d7fca440ba9d6627eac353f3f2036b5ad47bb2b WatchSource:0}: Error finding container a7607af822c42795b92e235e6d7fca440ba9d6627eac353f3f2036b5ad47bb2b: Status 404 returned error can't find the container with id a7607af822c42795b92e235e6d7fca440ba9d6627eac353f3f2036b5ad47bb2b
Sep 15 19:23:56.696809 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:56.696771    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2" event={"ID":"d54436b5-9066-4656-97c5-04fb36340a09","Type":"ContainerStarted","Data":"3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a"}
Sep 15 19:23:56.697256 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:56.696821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2" event={"ID":"d54436b5-9066-4656-97c5-04fb36340a09","Type":"ContainerStarted","Data":"a7607af822c42795b92e235e6d7fca440ba9d6627eac353f3f2036b5ad47bb2b"}
Sep 15 19:23:56.697256 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:56.696843    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:23:56.715094 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:23:56.715047    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2" podStartSLOduration=1.715030984 podStartE2EDuration="1.715030984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:23:56.713491103 +0000 UTC m=+1995.569344385" watchObservedRunningTime="2026-09-15 19:23:56.715030984 +0000 UTC m=+1995.570884315"
Sep 15 19:24:02.831845 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:02.831788    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq" podUID="56813081-07ff-49d3-af15-f769dec56750" containerName="main" containerID="cri-o://ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705" gracePeriod=45
Sep 15 19:24:03.088628 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.088604    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:24:03.200838 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.200804    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56813081-07ff-49d3-af15-f769dec56750-tls-certs\") pod \"56813081-07ff-49d3-af15-f769dec56750\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") "
Sep 15 19:24:03.200838 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.200839    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-home\") pod \"56813081-07ff-49d3-af15-f769dec56750\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") "
Sep 15 19:24:03.201104 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.200887    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78xzz\" (UniqueName: \"kubernetes.io/projected/56813081-07ff-49d3-af15-f769dec56750-kube-api-access-78xzz\") pod \"56813081-07ff-49d3-af15-f769dec56750\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") "
Sep 15 19:24:03.201104 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.200925    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-model-cache\") pod \"56813081-07ff-49d3-af15-f769dec56750\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") "
Sep 15 19:24:03.201104 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.200963    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-tmp-dir\") pod \"56813081-07ff-49d3-af15-f769dec56750\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") "
Sep 15 19:24:03.201104 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.201032    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-dshm\") pod \"56813081-07ff-49d3-af15-f769dec56750\" (UID: \"56813081-07ff-49d3-af15-f769dec56750\") "
Sep 15 19:24:03.201304 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.201170    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-model-cache" (OuterVolumeSpecName: "model-cache") pod "56813081-07ff-49d3-af15-f769dec56750" (UID: "56813081-07ff-49d3-af15-f769dec56750"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:24:03.201304 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.201202    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-home" (OuterVolumeSpecName: "home") pod "56813081-07ff-49d3-af15-f769dec56750" (UID: "56813081-07ff-49d3-af15-f769dec56750"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:24:03.201304 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.201281    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56813081-07ff-49d3-af15-f769dec56750" (UID: "56813081-07ff-49d3-af15-f769dec56750"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:24:03.201424 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.201360    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-home\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:24:03.201424 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.201379    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-model-cache\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:24:03.203212 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.203190    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56813081-07ff-49d3-af15-f769dec56750-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56813081-07ff-49d3-af15-f769dec56750" (UID: "56813081-07ff-49d3-af15-f769dec56750"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:24:03.203303 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.203273    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-dshm" (OuterVolumeSpecName: "dshm") pod "56813081-07ff-49d3-af15-f769dec56750" (UID: "56813081-07ff-49d3-af15-f769dec56750"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:24:03.203303 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.203294    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56813081-07ff-49d3-af15-f769dec56750-kube-api-access-78xzz" (OuterVolumeSpecName: "kube-api-access-78xzz") pod "56813081-07ff-49d3-af15-f769dec56750" (UID: "56813081-07ff-49d3-af15-f769dec56750"). InnerVolumeSpecName "kube-api-access-78xzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:24:03.302124 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.302086    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56813081-07ff-49d3-af15-f769dec56750-tls-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:24:03.302124 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.302115    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xzz\" (UniqueName: \"kubernetes.io/projected/56813081-07ff-49d3-af15-f769dec56750-kube-api-access-78xzz\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:24:03.302124 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.302125    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-tmp-dir\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:24:03.302340 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.302137    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56813081-07ff-49d3-af15-f769dec56750-dshm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:24:03.733108 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.733067    2556 generic.go:356] "Generic (PLEG): container finished" podID="56813081-07ff-49d3-af15-f769dec56750" containerID="ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705" exitCode=0
Sep 15 19:24:03.733290 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.733124    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"
Sep 15 19:24:03.733290 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.733157    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq" event={"ID":"56813081-07ff-49d3-af15-f769dec56750","Type":"ContainerDied","Data":"ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705"}
Sep 15 19:24:03.733290 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.733208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq" event={"ID":"56813081-07ff-49d3-af15-f769dec56750","Type":"ContainerDied","Data":"1f14cce8444e309ea865f88beb0d2ccc87241232fc39eece4445665c0c284076"}
Sep 15 19:24:03.733290 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.733231    2556 scope.go:117] "RemoveContainer" containerID="ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705"
Sep 15 19:24:03.742228 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.742206    2556 scope.go:117] "RemoveContainer" containerID="ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705"
Sep 15 19:24:03.742543 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:24:03.742518    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705\": container with ID starting with ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705 not found: ID does not exist" containerID="ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705"
Sep 15 19:24:03.742595 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.742556    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705"} err="failed to get container status \"ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705\": rpc error: code = NotFound desc = could not find container \"ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705\": container with ID starting with ce034ae2e17389374fcf9543fd63220d01910931c3d17334870155c5c56a4705 not found: ID does not exist"
Sep 15 19:24:03.752634 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.752609    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"]
Sep 15 19:24:03.755672 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:03.755626    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/rate-test-model-c6d10a-kserve-f66bd74dc-9w9sq"]
Sep 15 19:24:05.678129 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:05.678098    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56813081-07ff-49d3-af15-f769dec56750" path="/var/lib/kubelet/pods/56813081-07ff-49d3-af15-f769dec56750/volumes"
Sep 15 19:24:07.704152 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:07.704120    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:24:18.672093 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:18.672058    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:24:25.165991 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.165959    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"]
Sep 15 19:24:25.166419 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.166406    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56813081-07ff-49d3-af15-f769dec56750" containerName="main"
Sep 15 19:24:25.166419 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.166420    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="56813081-07ff-49d3-af15-f769dec56750" containerName="main"
Sep 15 19:24:25.166496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.166483    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="56813081-07ff-49d3-af15-f769dec56750" containerName="main"
Sep 15 19:24:25.169308 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.169285    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.171872 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.171848    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ce162-dockercfg-nbx8d\""
Sep 15 19:24:25.172156 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.172134    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ce162\""
Sep 15 19:24:25.179324 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.179301    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"]
Sep 15 19:24:25.207138 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.207103    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2hx\" (UniqueName: \"kubernetes.io/projected/90f6fbf3-c06f-406d-8862-630642cbba8b-kube-api-access-rz2hx\") pod \"payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq\" (UID: \"90f6fbf3-c06f-406d-8862-630642cbba8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.207304 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.207284    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f6fbf3-c06f-406d-8862-630642cbba8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq\" (UID: \"90f6fbf3-c06f-406d-8862-630642cbba8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.208078 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.208056    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"]
Sep 15 19:24:25.211863 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.211846    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:25.222763 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.222736    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"]
Sep 15 19:24:25.308693 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.308661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/7a7dfe72-7806-4524-b33f-90faad0afa17-kube-api-access-m8hnp\") pod \"payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs\" (UID: \"7a7dfe72-7806-4524-b33f-90faad0afa17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:25.308877 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.308709    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a7dfe72-7806-4524-b33f-90faad0afa17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs\" (UID: \"7a7dfe72-7806-4524-b33f-90faad0afa17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:25.308877 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.308746    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f6fbf3-c06f-406d-8862-630642cbba8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq\" (UID: \"90f6fbf3-c06f-406d-8862-630642cbba8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.308877 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.308786    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2hx\" (UniqueName: \"kubernetes.io/projected/90f6fbf3-c06f-406d-8862-630642cbba8b-kube-api-access-rz2hx\") pod \"payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq\" (UID: \"90f6fbf3-c06f-406d-8862-630642cbba8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.309803 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.309772    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f6fbf3-c06f-406d-8862-630642cbba8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq\" (UID: \"90f6fbf3-c06f-406d-8862-630642cbba8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.316537 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.316514    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2hx\" (UniqueName: \"kubernetes.io/projected/90f6fbf3-c06f-406d-8862-630642cbba8b-kube-api-access-rz2hx\") pod \"payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq\" (UID: \"90f6fbf3-c06f-406d-8862-630642cbba8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.410322 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.410288    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/7a7dfe72-7806-4524-b33f-90faad0afa17-kube-api-access-m8hnp\") pod \"payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs\" (UID: \"7a7dfe72-7806-4524-b33f-90faad0afa17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:25.410497 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.410346    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a7dfe72-7806-4524-b33f-90faad0afa17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs\" (UID: \"7a7dfe72-7806-4524-b33f-90faad0afa17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:25.411136 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.411108    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a7dfe72-7806-4524-b33f-90faad0afa17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs\" (UID: \"7a7dfe72-7806-4524-b33f-90faad0afa17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:25.417796 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.417742    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/7a7dfe72-7806-4524-b33f-90faad0afa17-kube-api-access-m8hnp\") pod \"payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs\" (UID: \"7a7dfe72-7806-4524-b33f-90faad0afa17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:25.480768 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.480740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.522610 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.522581    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:25.752440 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.752396    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"]
Sep 15 19:24:25.754527 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:24:25.754495    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f6fbf3_c06f_406d_8862_630642cbba8b.slice/crio-ce2e629a096d1b264bf9bb50612af96479553d9125ff9f576896dcba1408fc4d WatchSource:0}: Error finding container ce2e629a096d1b264bf9bb50612af96479553d9125ff9f576896dcba1408fc4d: Status 404 returned error can't find the container with id ce2e629a096d1b264bf9bb50612af96479553d9125ff9f576896dcba1408fc4d
Sep 15 19:24:25.801146 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.801123    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"]
Sep 15 19:24:25.803492 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:24:25.803466    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7dfe72_7806_4524_b33f_90faad0afa17.slice/crio-57a812054ddb65a21ff30e11cbf0e9fd98e7d560d27761bd1d64de180b41f82d WatchSource:0}: Error finding container 57a812054ddb65a21ff30e11cbf0e9fd98e7d560d27761bd1d64de180b41f82d: Status 404 returned error can't find the container with id 57a812054ddb65a21ff30e11cbf0e9fd98e7d560d27761bd1d64de180b41f82d
Sep 15 19:24:25.855886 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.855859    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs" event={"ID":"7a7dfe72-7806-4524-b33f-90faad0afa17","Type":"ContainerStarted","Data":"57a812054ddb65a21ff30e11cbf0e9fd98e7d560d27761bd1d64de180b41f82d"}
Sep 15 19:24:25.857581 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.857553    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq" event={"ID":"90f6fbf3-c06f-406d-8862-630642cbba8b","Type":"ContainerStarted","Data":"947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa"}
Sep 15 19:24:25.857733 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.857599    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq" event={"ID":"90f6fbf3-c06f-406d-8862-630642cbba8b","Type":"ContainerStarted","Data":"ce2e629a096d1b264bf9bb50612af96479553d9125ff9f576896dcba1408fc4d"}
Sep 15 19:24:25.857733 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.857664    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:25.879449 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:25.879401    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq" podStartSLOduration=0.879387219 podStartE2EDuration="879.387219ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:24:25.877520827 +0000 UTC m=+2024.733374100" watchObservedRunningTime="2026-09-15 19:24:25.879387219 +0000 UTC m=+2024.735240486"
Sep 15 19:24:26.865003 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:26.864963    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs" event={"ID":"7a7dfe72-7806-4524-b33f-90faad0afa17","Type":"ContainerStarted","Data":"ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0"}
Sep 15 19:24:26.865471 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:26.865052    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:26.882823 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:26.882772    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs" podStartSLOduration=1.882757945 podStartE2EDuration="1.882757945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:24:26.880280015 +0000 UTC m=+2025.736133284" watchObservedRunningTime="2026-09-15 19:24:26.882757945 +0000 UTC m=+2025.738611213"
Sep 15 19:24:36.867369 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:36.867336    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:24:37.873099 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:37.873066    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:24:52.359168 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:52.359137    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-74969cc5f8-hnjvz_3b536038-6bba-4a1a-a625-415d0e8712d6/payload-processing/0.log"
Sep 15 19:24:52.471412 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:52.471382    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-74969cc5f8-hnjvz_3b536038-6bba-4a1a-a625-415d0e8712d6/payload-processing/0.log"
Sep 15 19:24:52.592056 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:24:52.592024    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs_7a7dfe72-7806-4524-b33f-90faad0afa17/payload-processing/0.log"
Sep 15 19:25:01.675198 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:01.675156    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:25:20.672297 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:20.672220    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:25:38.908359 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:38.908330    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-74969cc5f8-hnjvz_3b536038-6bba-4a1a-a625-415d0e8712d6/payload-processing/0.log"
Sep 15 19:25:39.033865 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:39.033834    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-74969cc5f8-hnjvz_3b536038-6bba-4a1a-a625-415d0e8712d6/payload-processing/0.log"
Sep 15 19:25:39.155720 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:39.155691    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs_7a7dfe72-7806-4524-b33f-90faad0afa17/payload-processing/0.log"
Sep 15 19:25:39.269861 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:39.269782    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs_7a7dfe72-7806-4524-b33f-90faad0afa17/payload-processing/0.log"
Sep 15 19:25:41.874762 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:41.874736    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:25:41.880302 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:41.880277    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:25:41.882465 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:41.882447    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:25:41.887754 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:41.887737    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:25:47.234569 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.234530    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"]
Sep 15 19:25:47.238247 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.238230    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:47.240800 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.240778    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f8b852\""
Sep 15 19:25:47.250818 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.250797    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"]
Sep 15 19:25:47.276076 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.276043    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"]
Sep 15 19:25:47.280435 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.280415    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:47.282956 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.282932    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f8b852-dockercfg-6cnzj\""
Sep 15 19:25:47.287899 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.287872    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx\" (UID: \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:47.288022 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.287916    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhj9l\" (UniqueName: \"kubernetes.io/projected/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-kube-api-access-hhj9l\") pod \"payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx\" (UID: \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:47.288092 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.288033    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64v47\" (UniqueName: \"kubernetes.io/projected/3241373b-1708-4d3b-bf01-999734b86add-kube-api-access-64v47\") pod \"payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7\" (UID: \"3241373b-1708-4d3b-bf01-999734b86add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:47.288161 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.288120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3241373b-1708-4d3b-bf01-999734b86add-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7\" (UID: \"3241373b-1708-4d3b-bf01-999734b86add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:47.294176 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.294151    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"]
Sep 15 19:25:47.389046 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.389015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64v47\" (UniqueName: \"kubernetes.io/projected/3241373b-1708-4d3b-bf01-999734b86add-kube-api-access-64v47\") pod \"payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7\" (UID: \"3241373b-1708-4d3b-bf01-999734b86add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:47.389217 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.389068    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3241373b-1708-4d3b-bf01-999734b86add-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7\" (UID: \"3241373b-1708-4d3b-bf01-999734b86add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:47.389217 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.389111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx\" (UID: \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:47.389217 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.389130    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhj9l\" (UniqueName: \"kubernetes.io/projected/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-kube-api-access-hhj9l\") pod \"payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx\" (UID: \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:47.390002 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.389932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3241373b-1708-4d3b-bf01-999734b86add-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7\" (UID: \"3241373b-1708-4d3b-bf01-999734b86add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:47.390002 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.389937    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx\" (UID: \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:47.400291 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.400268    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64v47\" (UniqueName: \"kubernetes.io/projected/3241373b-1708-4d3b-bf01-999734b86add-kube-api-access-64v47\") pod \"payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7\" (UID: \"3241373b-1708-4d3b-bf01-999734b86add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:47.400484 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.400444    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhj9l\" (UniqueName: \"kubernetes.io/projected/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-kube-api-access-hhj9l\") pod \"payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx\" (UID: \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:47.550053 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.549961    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:47.593770 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.593732    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:47.827086 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.827054    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"]
Sep 15 19:25:47.828159 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:25:47.828135    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3241373b_1708_4d3b_bf01_999734b86add.slice/crio-43bdd5158d366549bedd5d0fc3fc821a70f6010fa3324642121ba3d4c3649023 WatchSource:0}: Error finding container 43bdd5158d366549bedd5d0fc3fc821a70f6010fa3324642121ba3d4c3649023: Status 404 returned error can't find the container with id 43bdd5158d366549bedd5d0fc3fc821a70f6010fa3324642121ba3d4c3649023
Sep 15 19:25:47.894723 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:47.894691    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"]
Sep 15 19:25:47.895612 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:25:47.895586    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a64801e_b78f_4e5f_bbe1_146d5dd12f50.slice/crio-78a5202c6b064a4371efabeefac4739b8c42ba0aef795977a17cff4e1dd52bf1 WatchSource:0}: Error finding container 78a5202c6b064a4371efabeefac4739b8c42ba0aef795977a17cff4e1dd52bf1: Status 404 returned error can't find the container with id 78a5202c6b064a4371efabeefac4739b8c42ba0aef795977a17cff4e1dd52bf1
Sep 15 19:25:48.297324 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:48.297284    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx" event={"ID":"6a64801e-b78f-4e5f-bbe1-146d5dd12f50","Type":"ContainerStarted","Data":"c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9"}
Sep 15 19:25:48.297782 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:48.297347    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx" event={"ID":"6a64801e-b78f-4e5f-bbe1-146d5dd12f50","Type":"ContainerStarted","Data":"78a5202c6b064a4371efabeefac4739b8c42ba0aef795977a17cff4e1dd52bf1"}
Sep 15 19:25:48.297782 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:48.297379    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:25:48.299063 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:48.299040    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7" event={"ID":"3241373b-1708-4d3b-bf01-999734b86add","Type":"ContainerStarted","Data":"f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd"}
Sep 15 19:25:48.299185 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:48.299093    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:48.299185 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:48.299119    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7" event={"ID":"3241373b-1708-4d3b-bf01-999734b86add","Type":"ContainerStarted","Data":"43bdd5158d366549bedd5d0fc3fc821a70f6010fa3324642121ba3d4c3649023"}
Sep 15 19:25:48.331337 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:48.331282    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx" podStartSLOduration=1.331263629 podStartE2EDuration="1.331263629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:25:48.327895866 +0000 UTC m=+2107.183749146" watchObservedRunningTime="2026-09-15 19:25:48.331263629 +0000 UTC m=+2107.187116898"
Sep 15 19:25:48.359844 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:48.359792    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7" podStartSLOduration=1.359773732 podStartE2EDuration="1.359773732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:25:48.357825113 +0000 UTC m=+2107.213678382" watchObservedRunningTime="2026-09-15 19:25:48.359773732 +0000 UTC m=+2107.215627003"
Sep 15 19:25:59.306847 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:59.306812    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:25:59.307264 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:25:59.306992    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:26:06.672270 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:06.672235    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:26:25.113702 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.113670    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"]
Sep 15 19:26:25.114279 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.114229    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx" podUID="6a64801e-b78f-4e5f-bbe1-146d5dd12f50" containerName="payload-processing" containerID="cri-o://c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9" gracePeriod=30
Sep 15 19:26:25.123469 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.123436    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"]
Sep 15 19:26:25.123799 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.123729    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7" podUID="3241373b-1708-4d3b-bf01-999734b86add" containerName="payload-pre-processing" containerID="cri-o://f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd" gracePeriod=30
Sep 15 19:26:25.378779 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.378753    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:26:25.412389 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.412364    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:26:25.498546 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.498508    2556 generic.go:356] "Generic (PLEG): container finished" podID="6a64801e-b78f-4e5f-bbe1-146d5dd12f50" containerID="c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9" exitCode=0
Sep 15 19:26:25.498546 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.498544    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"
Sep 15 19:26:25.498811 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.498590    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx" event={"ID":"6a64801e-b78f-4e5f-bbe1-146d5dd12f50","Type":"ContainerDied","Data":"c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9"}
Sep 15 19:26:25.498811 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.498667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx" event={"ID":"6a64801e-b78f-4e5f-bbe1-146d5dd12f50","Type":"ContainerDied","Data":"78a5202c6b064a4371efabeefac4739b8c42ba0aef795977a17cff4e1dd52bf1"}
Sep 15 19:26:25.498811 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.498689    2556 scope.go:117] "RemoveContainer" containerID="c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9"
Sep 15 19:26:25.500021 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.499997    2556 generic.go:356] "Generic (PLEG): container finished" podID="3241373b-1708-4d3b-bf01-999734b86add" containerID="f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd" exitCode=0
Sep 15 19:26:25.500121 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.500033    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"
Sep 15 19:26:25.500121 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.500051    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7" event={"ID":"3241373b-1708-4d3b-bf01-999734b86add","Type":"ContainerDied","Data":"f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd"}
Sep 15 19:26:25.500121 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.500089    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7" event={"ID":"3241373b-1708-4d3b-bf01-999734b86add","Type":"ContainerDied","Data":"43bdd5158d366549bedd5d0fc3fc821a70f6010fa3324642121ba3d4c3649023"}
Sep 15 19:26:25.507930 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.507913    2556 scope.go:117] "RemoveContainer" containerID="c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9"
Sep 15 19:26:25.508174 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:26:25.508157    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9\": container with ID starting with c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9 not found: ID does not exist" containerID="c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9"
Sep 15 19:26:25.508241 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.508181    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9"} err="failed to get container status \"c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9\": rpc error: code = NotFound desc = could not find container \"c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9\": container with ID starting with c3aecb0323b219677ac3462e51f9ca3593dde9d463a2ac7a3da1ebdc80a973b9 not found: ID does not exist"
Sep 15 19:26:25.508241 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.508205    2556 scope.go:117] "RemoveContainer" containerID="f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd"
Sep 15 19:26:25.516083 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.516063    2556 scope.go:117] "RemoveContainer" containerID="f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd"
Sep 15 19:26:25.516361 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:26:25.516342    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd\": container with ID starting with f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd not found: ID does not exist" containerID="f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd"
Sep 15 19:26:25.516420 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.516366    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd"} err="failed to get container status \"f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd\": rpc error: code = NotFound desc = could not find container \"f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd\": container with ID starting with f408ef8403810fcb0d0e64c910e033bb6ab4492c0e0141670c5cfd587fdc97bd not found: ID does not exist"
Sep 15 19:26:25.551329 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.551305    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-plugins-config-volume\") pod \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\" (UID: \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\") "
Sep 15 19:26:25.551428 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.551419    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3241373b-1708-4d3b-bf01-999734b86add-plugins-config-volume\") pod \"3241373b-1708-4d3b-bf01-999734b86add\" (UID: \"3241373b-1708-4d3b-bf01-999734b86add\") "
Sep 15 19:26:25.551478 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.551455    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhj9l\" (UniqueName: \"kubernetes.io/projected/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-kube-api-access-hhj9l\") pod \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\" (UID: \"6a64801e-b78f-4e5f-bbe1-146d5dd12f50\") "
Sep 15 19:26:25.551525 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.551506    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64v47\" (UniqueName: \"kubernetes.io/projected/3241373b-1708-4d3b-bf01-999734b86add-kube-api-access-64v47\") pod \"3241373b-1708-4d3b-bf01-999734b86add\" (UID: \"3241373b-1708-4d3b-bf01-999734b86add\") "
Sep 15 19:26:25.551791 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.551755    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a64801e-b78f-4e5f-bbe1-146d5dd12f50" (UID: "6a64801e-b78f-4e5f-bbe1-146d5dd12f50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:25.551886 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.551845    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3241373b-1708-4d3b-bf01-999734b86add-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3241373b-1708-4d3b-bf01-999734b86add" (UID: "3241373b-1708-4d3b-bf01-999734b86add"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:25.552007 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.551984    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3241373b-1708-4d3b-bf01-999734b86add-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:26:25.552007 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.552006    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:26:25.553589 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.553563    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-kube-api-access-hhj9l" (OuterVolumeSpecName: "kube-api-access-hhj9l") pod "6a64801e-b78f-4e5f-bbe1-146d5dd12f50" (UID: "6a64801e-b78f-4e5f-bbe1-146d5dd12f50"). InnerVolumeSpecName "kube-api-access-hhj9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:26:25.553752 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.553718    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3241373b-1708-4d3b-bf01-999734b86add-kube-api-access-64v47" (OuterVolumeSpecName: "kube-api-access-64v47") pod "3241373b-1708-4d3b-bf01-999734b86add" (UID: "3241373b-1708-4d3b-bf01-999734b86add"). InnerVolumeSpecName "kube-api-access-64v47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:26:25.653481 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.653398    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhj9l\" (UniqueName: \"kubernetes.io/projected/6a64801e-b78f-4e5f-bbe1-146d5dd12f50-kube-api-access-hhj9l\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:26:25.653481 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.653427    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64v47\" (UniqueName: \"kubernetes.io/projected/3241373b-1708-4d3b-bf01-999734b86add-kube-api-access-64v47\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:26:25.814237 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.814205    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"]
Sep 15 19:26:25.817871 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.817844    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f8b852-54f874d445-q89mx"]
Sep 15 19:26:25.828220 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.828194    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"]
Sep 15 19:26:25.830346 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:25.830328    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f8b852-845d98fcc4-6whr7"]
Sep 15 19:26:27.684009 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:27.683964    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3241373b-1708-4d3b-bf01-999734b86add" path="/var/lib/kubelet/pods/3241373b-1708-4d3b-bf01-999734b86add/volumes"
Sep 15 19:26:27.684632 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:27.684609    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a64801e-b78f-4e5f-bbe1-146d5dd12f50" path="/var/lib/kubelet/pods/6a64801e-b78f-4e5f-bbe1-146d5dd12f50/volumes"
Sep 15 19:26:37.672898 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:37.672825    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:26:50.827962 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:50.827931    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"]
Sep 15 19:26:50.828584 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:50.828529    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2" podUID="d54436b5-9066-4656-97c5-04fb36340a09" containerName="payload-pre-processing" containerID="cri-o://3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a" gracePeriod=30
Sep 15 19:26:51.091502 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.091446    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:26:51.196297 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.196264    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88dcl\" (UniqueName: \"kubernetes.io/projected/d54436b5-9066-4656-97c5-04fb36340a09-kube-api-access-88dcl\") pod \"d54436b5-9066-4656-97c5-04fb36340a09\" (UID: \"d54436b5-9066-4656-97c5-04fb36340a09\") "
Sep 15 19:26:51.196297 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.196304    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54436b5-9066-4656-97c5-04fb36340a09-plugins-config-volume\") pod \"d54436b5-9066-4656-97c5-04fb36340a09\" (UID: \"d54436b5-9066-4656-97c5-04fb36340a09\") "
Sep 15 19:26:51.196748 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.196723    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54436b5-9066-4656-97c5-04fb36340a09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d54436b5-9066-4656-97c5-04fb36340a09" (UID: "d54436b5-9066-4656-97c5-04fb36340a09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:51.198453 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.198430    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54436b5-9066-4656-97c5-04fb36340a09-kube-api-access-88dcl" (OuterVolumeSpecName: "kube-api-access-88dcl") pod "d54436b5-9066-4656-97c5-04fb36340a09" (UID: "d54436b5-9066-4656-97c5-04fb36340a09"). InnerVolumeSpecName "kube-api-access-88dcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:26:51.297539 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.297509    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88dcl\" (UniqueName: \"kubernetes.io/projected/d54436b5-9066-4656-97c5-04fb36340a09-kube-api-access-88dcl\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:26:51.297539 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.297534    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54436b5-9066-4656-97c5-04fb36340a09-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:26:51.651479 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.651444    2556 generic.go:356] "Generic (PLEG): container finished" podID="d54436b5-9066-4656-97c5-04fb36340a09" containerID="3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a" exitCode=0
Sep 15 19:26:51.651668 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.651490    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2" event={"ID":"d54436b5-9066-4656-97c5-04fb36340a09","Type":"ContainerDied","Data":"3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a"}
Sep 15 19:26:51.651668 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.651532    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2" event={"ID":"d54436b5-9066-4656-97c5-04fb36340a09","Type":"ContainerDied","Data":"a7607af822c42795b92e235e6d7fca440ba9d6627eac353f3f2036b5ad47bb2b"}
Sep 15 19:26:51.651668 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.651556    2556 scope.go:117] "RemoveContainer" containerID="3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a"
Sep 15 19:26:51.651668 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.651492    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"
Sep 15 19:26:51.661461 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.661442    2556 scope.go:117] "RemoveContainer" containerID="3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a"
Sep 15 19:26:51.661780 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:26:51.661759    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a\": container with ID starting with 3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a not found: ID does not exist" containerID="3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a"
Sep 15 19:26:51.661868 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.661790    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a"} err="failed to get container status \"3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a\": rpc error: code = NotFound desc = could not find container \"3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a\": container with ID starting with 3639b273624201edfd04602b2a786b1b13ae5b4cdbb9b44c77cf1ea3e24a532a not found: ID does not exist"
Sep 15 19:26:51.687928 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.687897    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"]
Sep 15 19:26:51.688082 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:51.687943    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f61404-6ddbbdbc74-zhrc2"]
Sep 15 19:26:53.679281 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:26:53.679245    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54436b5-9066-4656-97c5-04fb36340a09" path="/var/lib/kubelet/pods/d54436b5-9066-4656-97c5-04fb36340a09/volumes"
Sep 15 19:27:31.674518 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:31.674480    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:27:49.970892 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:49.970860    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"]
Sep 15 19:27:49.971373 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:49.971094    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs" podUID="7a7dfe72-7806-4524-b33f-90faad0afa17" containerName="payload-processing" containerID="cri-o://ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0" gracePeriod=30
Sep 15 19:27:49.976530 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:49.976501    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"]
Sep 15 19:27:49.976792 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:49.976746    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq" podUID="90f6fbf3-c06f-406d-8862-630642cbba8b" containerName="payload-pre-processing" containerID="cri-o://947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa" gracePeriod=30
Sep 15 19:27:50.270436 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.270413    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:27:50.273375 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.273344    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:27:50.349850 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.349817    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz2hx\" (UniqueName: \"kubernetes.io/projected/90f6fbf3-c06f-406d-8862-630642cbba8b-kube-api-access-rz2hx\") pod \"90f6fbf3-c06f-406d-8862-630642cbba8b\" (UID: \"90f6fbf3-c06f-406d-8862-630642cbba8b\") "
Sep 15 19:27:50.350034 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.349961    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a7dfe72-7806-4524-b33f-90faad0afa17-plugins-config-volume\") pod \"7a7dfe72-7806-4524-b33f-90faad0afa17\" (UID: \"7a7dfe72-7806-4524-b33f-90faad0afa17\") "
Sep 15 19:27:50.350034 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.349982    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/7a7dfe72-7806-4524-b33f-90faad0afa17-kube-api-access-m8hnp\") pod \"7a7dfe72-7806-4524-b33f-90faad0afa17\" (UID: \"7a7dfe72-7806-4524-b33f-90faad0afa17\") "
Sep 15 19:27:50.350034 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.350005    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f6fbf3-c06f-406d-8862-630642cbba8b-plugins-config-volume\") pod \"90f6fbf3-c06f-406d-8862-630642cbba8b\" (UID: \"90f6fbf3-c06f-406d-8862-630642cbba8b\") "
Sep 15 19:27:50.350437 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.350407    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7dfe72-7806-4524-b33f-90faad0afa17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a7dfe72-7806-4524-b33f-90faad0afa17" (UID: "7a7dfe72-7806-4524-b33f-90faad0afa17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:27:50.350606 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.350475    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f6fbf3-c06f-406d-8862-630642cbba8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90f6fbf3-c06f-406d-8862-630642cbba8b" (UID: "90f6fbf3-c06f-406d-8862-630642cbba8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:27:50.352240 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.352213    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f6fbf3-c06f-406d-8862-630642cbba8b-kube-api-access-rz2hx" (OuterVolumeSpecName: "kube-api-access-rz2hx") pod "90f6fbf3-c06f-406d-8862-630642cbba8b" (UID: "90f6fbf3-c06f-406d-8862-630642cbba8b"). InnerVolumeSpecName "kube-api-access-rz2hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:50.352357 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.352234    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7dfe72-7806-4524-b33f-90faad0afa17-kube-api-access-m8hnp" (OuterVolumeSpecName: "kube-api-access-m8hnp") pod "7a7dfe72-7806-4524-b33f-90faad0afa17" (UID: "7a7dfe72-7806-4524-b33f-90faad0afa17"). InnerVolumeSpecName "kube-api-access-m8hnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:50.451737 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.451698    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a7dfe72-7806-4524-b33f-90faad0afa17-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.451737 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.451734    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/7a7dfe72-7806-4524-b33f-90faad0afa17-kube-api-access-m8hnp\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.451925 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.451751    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f6fbf3-c06f-406d-8862-630642cbba8b-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.451925 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.451765    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz2hx\" (UniqueName: \"kubernetes.io/projected/90f6fbf3-c06f-406d-8862-630642cbba8b-kube-api-access-rz2hx\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.957993 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.956296    2556 generic.go:356] "Generic (PLEG): container finished" podID="90f6fbf3-c06f-406d-8862-630642cbba8b" containerID="947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa" exitCode=0
Sep 15 19:27:50.957993 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.956475    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq" event={"ID":"90f6fbf3-c06f-406d-8862-630642cbba8b","Type":"ContainerDied","Data":"947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa"}
Sep 15 19:27:50.957993 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.956527    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq" event={"ID":"90f6fbf3-c06f-406d-8862-630642cbba8b","Type":"ContainerDied","Data":"ce2e629a096d1b264bf9bb50612af96479553d9125ff9f576896dcba1408fc4d"}
Sep 15 19:27:50.957993 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.956555    2556 scope.go:117] "RemoveContainer" containerID="947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa"
Sep 15 19:27:50.957993 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.956753    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"
Sep 15 19:27:50.959702 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.959673    2556 generic.go:356] "Generic (PLEG): container finished" podID="7a7dfe72-7806-4524-b33f-90faad0afa17" containerID="ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0" exitCode=0
Sep 15 19:27:50.959831 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.959752    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs" event={"ID":"7a7dfe72-7806-4524-b33f-90faad0afa17","Type":"ContainerDied","Data":"ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0"}
Sep 15 19:27:50.959831 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.959740    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"
Sep 15 19:27:50.959831 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.959822    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs" event={"ID":"7a7dfe72-7806-4524-b33f-90faad0afa17","Type":"ContainerDied","Data":"57a812054ddb65a21ff30e11cbf0e9fd98e7d560d27761bd1d64de180b41f82d"}
Sep 15 19:27:50.970028 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.970010    2556 scope.go:117] "RemoveContainer" containerID="947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa"
Sep 15 19:27:50.970436 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:27:50.970325    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa\": container with ID starting with 947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa not found: ID does not exist" containerID="947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa"
Sep 15 19:27:50.970436 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.970360    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa"} err="failed to get container status \"947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa\": rpc error: code = NotFound desc = could not find container \"947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa\": container with ID starting with 947dad8028375e10c5a98897462eda961eda014542adde2413d45afc23ebaaaa not found: ID does not exist"
Sep 15 19:27:50.970436 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.970385    2556 scope.go:117] "RemoveContainer" containerID="ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0"
Sep 15 19:27:50.979551 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.979528    2556 scope.go:117] "RemoveContainer" containerID="ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0"
Sep 15 19:27:50.980201 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:27:50.979932    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0\": container with ID starting with ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0 not found: ID does not exist" containerID="ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0"
Sep 15 19:27:50.980201 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.979964    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0"} err="failed to get container status \"ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0\": rpc error: code = NotFound desc = could not find container \"ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0\": container with ID starting with ce98512f0ca7aa9b491bd0bf140bc42241d176217a766ec759a0f4044599a1c0 not found: ID does not exist"
Sep 15 19:27:50.981102 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.981082    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"]
Sep 15 19:27:50.982919 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.982898    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ce162-595fd87f6b-8c8wq"]
Sep 15 19:27:50.992109 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.992088    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"]
Sep 15 19:27:50.994703 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:50.994679    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce162-6977b9b9f7-8x2xs"]
Sep 15 19:27:51.678619 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:51.678585    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7dfe72-7806-4524-b33f-90faad0afa17" path="/var/lib/kubelet/pods/7a7dfe72-7806-4524-b33f-90faad0afa17/volumes"
Sep 15 19:27:51.679054 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:51.679032    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f6fbf3-c06f-406d-8862-630642cbba8b" path="/var/lib/kubelet/pods/90f6fbf3-c06f-406d-8862-630642cbba8b/volumes"
Sep 15 19:27:58.671744 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:27:58.671711    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:28:07.657554 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:07.657464    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"]
Sep 15 19:28:07.658000 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:07.657774    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k" podUID="e10ff7a4-0dbb-4959-bf99-ef542190ec08" containerName="payload-processing" containerID="cri-o://f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1" gracePeriod=30
Sep 15 19:28:07.911868 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:07.911813    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:28:08.018285 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.018257    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10ff7a4-0dbb-4959-bf99-ef542190ec08-plugins-config-volume\") pod \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\" (UID: \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\") "
Sep 15 19:28:08.018436 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.018363    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptjtm\" (UniqueName: \"kubernetes.io/projected/e10ff7a4-0dbb-4959-bf99-ef542190ec08-kube-api-access-ptjtm\") pod \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\" (UID: \"e10ff7a4-0dbb-4959-bf99-ef542190ec08\") "
Sep 15 19:28:08.018699 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.018678    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10ff7a4-0dbb-4959-bf99-ef542190ec08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e10ff7a4-0dbb-4959-bf99-ef542190ec08" (UID: "e10ff7a4-0dbb-4959-bf99-ef542190ec08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:08.020189 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.020159    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10ff7a4-0dbb-4959-bf99-ef542190ec08-kube-api-access-ptjtm" (OuterVolumeSpecName: "kube-api-access-ptjtm") pod "e10ff7a4-0dbb-4959-bf99-ef542190ec08" (UID: "e10ff7a4-0dbb-4959-bf99-ef542190ec08"). InnerVolumeSpecName "kube-api-access-ptjtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:08.047993 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.047961    2556 generic.go:356] "Generic (PLEG): container finished" podID="e10ff7a4-0dbb-4959-bf99-ef542190ec08" containerID="f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1" exitCode=0
Sep 15 19:28:08.048131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.048011    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"
Sep 15 19:28:08.048131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.048053    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k" event={"ID":"e10ff7a4-0dbb-4959-bf99-ef542190ec08","Type":"ContainerDied","Data":"f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1"}
Sep 15 19:28:08.048131 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.048109    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k" event={"ID":"e10ff7a4-0dbb-4959-bf99-ef542190ec08","Type":"ContainerDied","Data":"a8bdf00e44a3cc9ff153e58147b8433a165b7f3717be7c99ed960560df3a0147"}
Sep 15 19:28:08.048302 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.048137    2556 scope.go:117] "RemoveContainer" containerID="f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1"
Sep 15 19:28:08.058093 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.058061    2556 scope.go:117] "RemoveContainer" containerID="f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1"
Sep 15 19:28:08.058386 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:28:08.058366    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1\": container with ID starting with f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1 not found: ID does not exist" containerID="f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1"
Sep 15 19:28:08.058445 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.058394    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1"} err="failed to get container status \"f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1\": rpc error: code = NotFound desc = could not find container \"f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1\": container with ID starting with f56536f84b191e37354265a7c1da5393f01c63fc85635bb88f0dd5767c8b4ec1 not found: ID does not exist"
Sep 15 19:28:08.069226 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.069199    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"]
Sep 15 19:28:08.071011 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.070991    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c6d10a-d97dc75cf-fn84k"]
Sep 15 19:28:08.119804 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.119782    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjtm\" (UniqueName: \"kubernetes.io/projected/e10ff7a4-0dbb-4959-bf99-ef542190ec08-kube-api-access-ptjtm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:28:08.119804 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:08.119803    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10ff7a4-0dbb-4959-bf99-ef542190ec08-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:28:09.679437 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:09.679400    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10ff7a4-0dbb-4959-bf99-ef542190ec08" path="/var/lib/kubelet/pods/e10ff7a4-0dbb-4959-bf99-ef542190ec08/volumes"
Sep 15 19:28:25.117834 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.117800    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"]
Sep 15 19:28:25.118396 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.118031    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6" podUID="1ae98ae1-ea22-4b15-909b-68cdc73b2972" containerName="payload-processing" containerID="cri-o://e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2" gracePeriod=30
Sep 15 19:28:25.371537 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.371510    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:28:25.484803 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.484768    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae98ae1-ea22-4b15-909b-68cdc73b2972-plugins-config-volume\") pod \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\" (UID: \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\") "
Sep 15 19:28:25.485008 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.484921    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbb9j\" (UniqueName: \"kubernetes.io/projected/1ae98ae1-ea22-4b15-909b-68cdc73b2972-kube-api-access-mbb9j\") pod \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\" (UID: \"1ae98ae1-ea22-4b15-909b-68cdc73b2972\") "
Sep 15 19:28:25.485197 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.485169    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae98ae1-ea22-4b15-909b-68cdc73b2972-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ae98ae1-ea22-4b15-909b-68cdc73b2972" (UID: "1ae98ae1-ea22-4b15-909b-68cdc73b2972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:25.486994 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.486970    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae98ae1-ea22-4b15-909b-68cdc73b2972-kube-api-access-mbb9j" (OuterVolumeSpecName: "kube-api-access-mbb9j") pod "1ae98ae1-ea22-4b15-909b-68cdc73b2972" (UID: "1ae98ae1-ea22-4b15-909b-68cdc73b2972"). InnerVolumeSpecName "kube-api-access-mbb9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:25.586173 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.586142    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbb9j\" (UniqueName: \"kubernetes.io/projected/1ae98ae1-ea22-4b15-909b-68cdc73b2972-kube-api-access-mbb9j\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:28:25.586173 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:25.586171    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae98ae1-ea22-4b15-909b-68cdc73b2972-plugins-config-volume\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:28:26.147515 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.147474    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ae98ae1-ea22-4b15-909b-68cdc73b2972" containerID="e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2" exitCode=0
Sep 15 19:28:26.147515 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.147517    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"
Sep 15 19:28:26.147979 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.147510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6" event={"ID":"1ae98ae1-ea22-4b15-909b-68cdc73b2972","Type":"ContainerDied","Data":"e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2"}
Sep 15 19:28:26.147979 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.147656    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6" event={"ID":"1ae98ae1-ea22-4b15-909b-68cdc73b2972","Type":"ContainerDied","Data":"37ae875af2de0a425c73b53dc12d0cef0382df22aeefc1c7c6befb45f74cfe02"}
Sep 15 19:28:26.147979 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.147682    2556 scope.go:117] "RemoveContainer" containerID="e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2"
Sep 15 19:28:26.156832 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.156813    2556 scope.go:117] "RemoveContainer" containerID="e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2"
Sep 15 19:28:26.157083 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:28:26.157064    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2\": container with ID starting with e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2 not found: ID does not exist" containerID="e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2"
Sep 15 19:28:26.157142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.157093    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2"} err="failed to get container status \"e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2\": rpc error: code = NotFound desc = could not find container \"e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2\": container with ID starting with e7137587099548dd8f214993f48c6a1f5eb8193bada435b4961eb642ab486de2 not found: ID does not exist"
Sep 15 19:28:26.166132 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.166110    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"]
Sep 15 19:28:26.169721 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:26.169697    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-415f87-7c5c69c49-5jhp6"]
Sep 15 19:28:27.678477 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:27.678446    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae98ae1-ea22-4b15-909b-68cdc73b2972" path="/var/lib/kubelet/pods/1ae98ae1-ea22-4b15-909b-68cdc73b2972/volumes"
Sep 15 19:28:33.812421 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:33.812390    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-v74sg"]
Sep 15 19:28:33.812854 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:33.812587    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg" podUID="f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" containerName="manager" containerID="cri-o://fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f" gracePeriod=30
Sep 15 19:28:34.063836 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.063782    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:28:34.165354 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.165321    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-webhook-cert\") pod \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") "
Sep 15 19:28:34.165515 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.165367    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bqdk\" (UniqueName: \"kubernetes.io/projected/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-kube-api-access-9bqdk\") pod \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") "
Sep 15 19:28:34.165515 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.165439    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-metrics-certs\") pod \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\" (UID: \"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63\") "
Sep 15 19:28:34.167510 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.167479    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" (UID: "f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:28:34.167510 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.167497    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" (UID: "f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:28:34.167681 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.167527    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-kube-api-access-9bqdk" (OuterVolumeSpecName: "kube-api-access-9bqdk") pod "f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" (UID: "f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63"). InnerVolumeSpecName "kube-api-access-9bqdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:34.191237 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.191208    2556 generic.go:356] "Generic (PLEG): container finished" podID="f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" containerID="fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f" exitCode=0
Sep 15 19:28:34.191338 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.191254    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg"
Sep 15 19:28:34.191338 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.191292    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg" event={"ID":"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63","Type":"ContainerDied","Data":"fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f"}
Sep 15 19:28:34.191415 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.191344    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-v74sg" event={"ID":"f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63","Type":"ContainerDied","Data":"3d505d1ef43fdbe701f2f8be38a36b4dce5d722dd9033d680ce4a1cbd3a407fb"}
Sep 15 19:28:34.191415 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.191367    2556 scope.go:117] "RemoveContainer" containerID="fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f"
Sep 15 19:28:34.201014 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.200999    2556 scope.go:117] "RemoveContainer" containerID="fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f"
Sep 15 19:28:34.201278 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:28:34.201259    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f\": container with ID starting with fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f not found: ID does not exist" containerID="fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f"
Sep 15 19:28:34.201324 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.201285    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f"} err="failed to get container status \"fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f\": rpc error: code = NotFound desc = could not find container \"fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f\": container with ID starting with fd0e10c1f4e39eb1f3d01539ac1b909781d388216df40b8b328034f1ea1e0a0f not found: ID does not exist"
Sep 15 19:28:34.212566 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.212540    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-v74sg"]
Sep 15 19:28:34.214513 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.214490    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-v74sg"]
Sep 15 19:28:34.266878 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.266856    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:28:34.266878 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.266875    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-webhook-cert\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:28:34.267001 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:34.266884    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bqdk\" (UniqueName: \"kubernetes.io/projected/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63-kube-api-access-9bqdk\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:28:35.677652 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:35.677601    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" path="/var/lib/kubelet/pods/f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63/volumes"
Sep 15 19:28:36.672043 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:28:36.672011    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:29:27.672286 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:27.672247    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:29:34.992996 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.992918    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-jmm7k"]
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993358    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f6fbf3-c06f-406d-8862-630642cbba8b" containerName="payload-pre-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993373    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f6fbf3-c06f-406d-8862-630642cbba8b" containerName="payload-pre-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993382    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7dfe72-7806-4524-b33f-90faad0afa17" containerName="payload-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993387    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7dfe72-7806-4524-b33f-90faad0afa17" containerName="payload-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993403    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10ff7a4-0dbb-4959-bf99-ef542190ec08" containerName="payload-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993410    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10ff7a4-0dbb-4959-bf99-ef542190ec08" containerName="payload-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993416    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54436b5-9066-4656-97c5-04fb36340a09" containerName="payload-pre-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993423    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54436b5-9066-4656-97c5-04fb36340a09" containerName="payload-pre-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993436    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3241373b-1708-4d3b-bf01-999734b86add" containerName="payload-pre-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993442    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3241373b-1708-4d3b-bf01-999734b86add" containerName="payload-pre-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993448    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a64801e-b78f-4e5f-bbe1-146d5dd12f50" containerName="payload-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993453    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a64801e-b78f-4e5f-bbe1-146d5dd12f50" containerName="payload-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993458    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae98ae1-ea22-4b15-909b-68cdc73b2972" containerName="payload-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993464    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae98ae1-ea22-4b15-909b-68cdc73b2972" containerName="payload-processing"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993474    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" containerName="manager"
Sep 15 19:29:34.993496 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993480    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" containerName="manager"
Sep 15 19:29:34.994142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993536    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54436b5-9066-4656-97c5-04fb36340a09" containerName="payload-pre-processing"
Sep 15 19:29:34.994142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993545    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7dfe72-7806-4524-b33f-90faad0afa17" containerName="payload-processing"
Sep 15 19:29:34.994142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993551    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3241373b-1708-4d3b-bf01-999734b86add" containerName="payload-pre-processing"
Sep 15 19:29:34.994142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993557    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a64801e-b78f-4e5f-bbe1-146d5dd12f50" containerName="payload-processing"
Sep 15 19:29:34.994142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993563    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fc7cd2-1ac5-4f23-8e57-fabbc3d6fa63" containerName="manager"
Sep 15 19:29:34.994142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993570    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10ff7a4-0dbb-4959-bf99-ef542190ec08" containerName="payload-processing"
Sep 15 19:29:34.994142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993574    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f6fbf3-c06f-406d-8862-630642cbba8b" containerName="payload-pre-processing"
Sep 15 19:29:34.994142 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.993581    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae98ae1-ea22-4b15-909b-68cdc73b2972" containerName="payload-processing"
Sep 15 19:29:34.996611 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.996595    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:34.999820 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.999796    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 19:29:34.999954 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.999827    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vc5j9\""
Sep 15 19:29:34.999954 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.999831    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 19:29:34.999954 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:34.999828    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 19:29:35.006075 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.006052    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-jmm7k"]
Sep 15 19:29:35.114947 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.114915    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.115136 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.114984    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.115136 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.115012    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnln\" (UniqueName: \"kubernetes.io/projected/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-kube-api-access-zdnln\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.216101 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.216066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.216310 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.216119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnln\" (UniqueName: \"kubernetes.io/projected/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-kube-api-access-zdnln\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.216310 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.216203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.218737 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.218702    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.218845 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.218702    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.225325 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.225302    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnln\" (UniqueName: \"kubernetes.io/projected/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-kube-api-access-zdnln\") pod \"maas-controller-5c9bcfd8cf-jmm7k\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") " pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.308419 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.308353    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:35.582815 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.582788    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-jmm7k"]
Sep 15 19:29:35.585025 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:29:35.585000    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e866f5_0a5b_4ec8_8d2f_0d7bf2f31314.slice/crio-8ef2259dda8f16271378be55e16f2f0645bb4dafdaaf1f423b804f505546374c WatchSource:0}: Error finding container 8ef2259dda8f16271378be55e16f2f0645bb4dafdaaf1f423b804f505546374c: Status 404 returned error can't find the container with id 8ef2259dda8f16271378be55e16f2f0645bb4dafdaaf1f423b804f505546374c
Sep 15 19:29:35.586308 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:35.586291    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:29:36.500733 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:36.500693    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k" event={"ID":"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314","Type":"ContainerStarted","Data":"84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196"}
Sep 15 19:29:36.501216 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:36.500758    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:36.501216 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:36.500782    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k" event={"ID":"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314","Type":"ContainerStarted","Data":"8ef2259dda8f16271378be55e16f2f0645bb4dafdaaf1f423b804f505546374c"}
Sep 15 19:29:36.516430 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:36.516379    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k" podStartSLOduration=2.024868591 podStartE2EDuration="2.516364833s" totalImagesPullingTime="491.496242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:29:36.514468279 +0000 UTC m=+2335.370321547" watchObservedRunningTime="2026-09-15 19:29:36.516364833 +0000 UTC m=+2335.372218102"
Sep 15 19:29:47.510813 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:47.510777    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:54.672583 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:54.672551    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:29:56.797690 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:56.797650    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-jmm7k"]
Sep 15 19:29:56.798117 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:56.797850    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k" podUID="e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" containerName="manager" containerID="cri-o://84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196" gracePeriod=30
Sep 15 19:29:57.044061 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.044037    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:57.228308 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.228280    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-webhook-cert\") pod \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") "
Sep 15 19:29:57.228493 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.228345    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdnln\" (UniqueName: \"kubernetes.io/projected/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-kube-api-access-zdnln\") pod \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") "
Sep 15 19:29:57.228493 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.228456    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-metrics-certs\") pod \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\" (UID: \"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314\") "
Sep 15 19:29:57.230365 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.230332    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" (UID: "e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:29:57.230471 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.230388    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-kube-api-access-zdnln" (OuterVolumeSpecName: "kube-api-access-zdnln") pod "e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" (UID: "e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314"). InnerVolumeSpecName "kube-api-access-zdnln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:29:57.230471 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.230418    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" (UID: "e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:29:57.329664 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.329620    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-webhook-cert\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:29:57.329664 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.329664    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdnln\" (UniqueName: \"kubernetes.io/projected/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-kube-api-access-zdnln\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:29:57.329832 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.329696    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:29:57.612759 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.612668    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" containerID="84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196" exitCode=0
Sep 15 19:29:57.612759 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.612723    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k"
Sep 15 19:29:57.612982 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.612751    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k" event={"ID":"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314","Type":"ContainerDied","Data":"84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196"}
Sep 15 19:29:57.612982 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.612846    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-jmm7k" event={"ID":"e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314","Type":"ContainerDied","Data":"8ef2259dda8f16271378be55e16f2f0645bb4dafdaaf1f423b804f505546374c"}
Sep 15 19:29:57.612982 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.612877    2556 scope.go:117] "RemoveContainer" containerID="84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196"
Sep 15 19:29:57.622250 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.622227    2556 scope.go:117] "RemoveContainer" containerID="84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196"
Sep 15 19:29:57.622516 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:29:57.622494    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196\": container with ID starting with 84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196 not found: ID does not exist" containerID="84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196"
Sep 15 19:29:57.622582 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.622522    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196"} err="failed to get container status \"84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196\": rpc error: code = NotFound desc = could not find container \"84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196\": container with ID starting with 84eae0fb6fed8ddea1a4b6da6bf6428daf6634a9727d89b4eb5aeec44bbce196 not found: ID does not exist"
Sep 15 19:29:57.633388 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.633365    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-jmm7k"]
Sep 15 19:29:57.635766 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.635746    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-jmm7k"]
Sep 15 19:29:57.678100 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:57.678077    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" path="/var/lib/kubelet/pods/e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314/volumes"
Sep 15 19:29:58.417338 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.417303    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-4j4zc"]
Sep 15 19:29:58.417769 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.417753    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" containerName="manager"
Sep 15 19:29:58.417819 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.417771    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" containerName="manager"
Sep 15 19:29:58.417852 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.417831    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e866f5-0a5b-4ec8-8d2f-0d7bf2f31314" containerName="manager"
Sep 15 19:29:58.422295 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.422276    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.425918 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.425896    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 19:29:58.426045 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.425896    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vc5j9\""
Sep 15 19:29:58.426045 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.425982    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 19:29:58.426045 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.426037    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 19:29:58.431494 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.431471    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-4j4zc"]
Sep 15 19:29:58.541970 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.541934    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.541970 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.541972    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.542162 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.542052    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wp7\" (UniqueName: \"kubernetes.io/projected/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-kube-api-access-c2wp7\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.642660 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.642595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.642660 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.642636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.642922 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.642702    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wp7\" (UniqueName: \"kubernetes.io/projected/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-kube-api-access-c2wp7\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.645147 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.645125    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-metrics-certs\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.645278 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.645261    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-webhook-cert\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.649958 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.649938    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wp7\" (UniqueName: \"kubernetes.io/projected/049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd-kube-api-access-c2wp7\") pod \"maas-controller-5c9bcfd8cf-4j4zc\" (UID: \"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd\") " pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:58.734093 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:58.734024    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:29:59.005478 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:59.005445    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9bcfd8cf-4j4zc"]
Sep 15 19:29:59.007765 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:29:59.007740    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049f48e6_04a7_4f0a_b72c_ea1b0cdf86fd.slice/crio-88b7044b309fd64dcdcc271ed003309bcbf3e9861aa51a7be6b8f0340ee894af WatchSource:0}: Error finding container 88b7044b309fd64dcdcc271ed003309bcbf3e9861aa51a7be6b8f0340ee894af: Status 404 returned error can't find the container with id 88b7044b309fd64dcdcc271ed003309bcbf3e9861aa51a7be6b8f0340ee894af
Sep 15 19:29:59.625445 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:29:59.625407    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc" event={"ID":"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd","Type":"ContainerStarted","Data":"88b7044b309fd64dcdcc271ed003309bcbf3e9861aa51a7be6b8f0340ee894af"}
Sep 15 19:30:00.631698 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:00.631654    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc" event={"ID":"049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd","Type":"ContainerStarted","Data":"4fba275600991eac094d8585dfcf9714385795635f263da255bfe91268ddbccc"}
Sep 15 19:30:00.632068 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:00.631725    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:30:00.648454 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:00.648404    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc" podStartSLOduration=1.702586215 podStartE2EDuration="2.64838934s" totalImagesPullingTime="945.803125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:30:00.64686768 +0000 UTC m=+2359.502720948" watchObservedRunningTime="2026-09-15 19:30:00.64838934 +0000 UTC m=+2359.504242607"
Sep 15 19:30:11.639634 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:11.639601    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c9bcfd8cf-4j4zc"
Sep 15 19:30:41.908931 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:41.908823    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:30:41.914100 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:41.914077    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:30:41.917428 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:41.917411    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:30:41.922201 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:41.922183    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfh72_175a2bd9-d6b5-4f02-a5b5-650a0fa125f2/ovn-acl-logging/0.log"
Sep 15 19:30:48.672634 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:30:48.672602    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:31:12.671782 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:31:12.671708    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:32:04.672102 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:32:04.672068    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:32:40.671706 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:32:40.671609    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:33:13.671780 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:33:13.671746    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:33:45.672838 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:33:45.672804    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:34:17.364479 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:17.364379    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"]
Sep 15 19:34:17.365064 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:17.364674    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72" podUID="56dff1f4-d575-4103-870e-9de2dda61b67" containerName="maas-api" containerID="cri-o://32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5" gracePeriod=30
Sep 15 19:34:18.608633 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.608608    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:34:18.715682 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.715581    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56dff1f4-d575-4103-870e-9de2dda61b67-openshift-service-ca\") pod \"56dff1f4-d575-4103-870e-9de2dda61b67\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") "
Sep 15 19:34:18.715682 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.715623    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-metrics-certs\") pod \"56dff1f4-d575-4103-870e-9de2dda61b67\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") "
Sep 15 19:34:18.715879 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.715692    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-maas-api-tls\") pod \"56dff1f4-d575-4103-870e-9de2dda61b67\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") "
Sep 15 19:34:18.715879 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.715825    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz5dm\" (UniqueName: \"kubernetes.io/projected/56dff1f4-d575-4103-870e-9de2dda61b67-kube-api-access-mz5dm\") pod \"56dff1f4-d575-4103-870e-9de2dda61b67\" (UID: \"56dff1f4-d575-4103-870e-9de2dda61b67\") "
Sep 15 19:34:18.716038 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.716017    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dff1f4-d575-4103-870e-9de2dda61b67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56dff1f4-d575-4103-870e-9de2dda61b67" (UID: "56dff1f4-d575-4103-870e-9de2dda61b67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:34:18.716097 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.716085    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56dff1f4-d575-4103-870e-9de2dda61b67-openshift-service-ca\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:34:18.717828 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.717802    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56dff1f4-d575-4103-870e-9de2dda61b67" (UID: "56dff1f4-d575-4103-870e-9de2dda61b67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:34:18.717945 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.717798    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dff1f4-d575-4103-870e-9de2dda61b67-kube-api-access-mz5dm" (OuterVolumeSpecName: "kube-api-access-mz5dm") pod "56dff1f4-d575-4103-870e-9de2dda61b67" (UID: "56dff1f4-d575-4103-870e-9de2dda61b67"). InnerVolumeSpecName "kube-api-access-mz5dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:34:18.717945 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.717860    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56dff1f4-d575-4103-870e-9de2dda61b67" (UID: "56dff1f4-d575-4103-870e-9de2dda61b67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:34:18.817592 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.817564    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-metrics-certs\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:34:18.817592 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.817586    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56dff1f4-d575-4103-870e-9de2dda61b67-maas-api-tls\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:34:18.817592 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.817595    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz5dm\" (UniqueName: \"kubernetes.io/projected/56dff1f4-d575-4103-870e-9de2dda61b67-kube-api-access-mz5dm\") on node \"ip-10-0-140-183.ec2.internal\" DevicePath \"\""
Sep 15 19:34:18.918588 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.918555    2556 generic.go:356] "Generic (PLEG): container finished" podID="56dff1f4-d575-4103-870e-9de2dda61b67" containerID="32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5" exitCode=0
Sep 15 19:34:18.918794 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.918600    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"
Sep 15 19:34:18.918794 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.918653    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72" event={"ID":"56dff1f4-d575-4103-870e-9de2dda61b67","Type":"ContainerDied","Data":"32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5"}
Sep 15 19:34:18.918794 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.918727    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-95b72" event={"ID":"56dff1f4-d575-4103-870e-9de2dda61b67","Type":"ContainerDied","Data":"c3269ad1af1e2adb98d72f1681fb2b6ecee97034d4bdace546e59cb99f9cd82c"}
Sep 15 19:34:18.918794 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.918758    2556 scope.go:117] "RemoveContainer" containerID="32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5"
Sep 15 19:34:18.928424 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.928405    2556 scope.go:117] "RemoveContainer" containerID="32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5"
Sep 15 19:34:18.928702 ip-10-0-140-183 kubenswrapper[2556]: E0915 19:34:18.928683    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5\": container with ID starting with 32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5 not found: ID does not exist" containerID="32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5"
Sep 15 19:34:18.928775 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.928714    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5"} err="failed to get container status \"32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5\": rpc error: code = NotFound desc = could not find container \"32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5\": container with ID starting with 32e3c5d0900edb83fcf64223d9444c9bf75e6e335b99bdfc43164b8765c166a5 not found: ID does not exist"
Sep 15 19:34:18.940250 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.940224    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"]
Sep 15 19:34:18.941971 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:18.941941    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-95b72"]
Sep 15 19:34:19.679187 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:19.679150    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56dff1f4-d575-4103-870e-9de2dda61b67" path="/var/lib/kubelet/pods/56dff1f4-d575-4103-870e-9de2dda61b67/volumes"
Sep 15 19:34:28.922627 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.922593    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xxzwt/must-gather-k5rrr"]
Sep 15 19:34:28.923027 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.923006    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56dff1f4-d575-4103-870e-9de2dda61b67" containerName="maas-api"
Sep 15 19:34:28.923027 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.923019    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="56dff1f4-d575-4103-870e-9de2dda61b67" containerName="maas-api"
Sep 15 19:34:28.923103 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.923088    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="56dff1f4-d575-4103-870e-9de2dda61b67" containerName="maas-api"
Sep 15 19:34:28.926318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.926300    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxzwt/must-gather-k5rrr"
Sep 15 19:34:28.929311 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.929287    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xxzwt\"/\"kube-root-ca.crt\""
Sep 15 19:34:28.930331 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.930313    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xxzwt\"/\"default-dockercfg-7hxc4\""
Sep 15 19:34:28.930331 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.930321    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xxzwt\"/\"openshift-service-ca.crt\""
Sep 15 19:34:28.936589 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:28.936570    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxzwt/must-gather-k5rrr"]
Sep 15 19:34:29.010555 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.010526    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r66tg\" (UniqueName: \"kubernetes.io/projected/9b7d9cc0-9c8b-44c3-a848-fde069de28ac-kube-api-access-r66tg\") pod \"must-gather-k5rrr\" (UID: \"9b7d9cc0-9c8b-44c3-a848-fde069de28ac\") " pod="openshift-must-gather-xxzwt/must-gather-k5rrr"
Sep 15 19:34:29.010731 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.010564    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b7d9cc0-9c8b-44c3-a848-fde069de28ac-must-gather-output\") pod \"must-gather-k5rrr\" (UID: \"9b7d9cc0-9c8b-44c3-a848-fde069de28ac\") " pod="openshift-must-gather-xxzwt/must-gather-k5rrr"
Sep 15 19:34:29.111626 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.111599    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r66tg\" (UniqueName: \"kubernetes.io/projected/9b7d9cc0-9c8b-44c3-a848-fde069de28ac-kube-api-access-r66tg\") pod \"must-gather-k5rrr\" (UID: \"9b7d9cc0-9c8b-44c3-a848-fde069de28ac\") " pod="openshift-must-gather-xxzwt/must-gather-k5rrr"
Sep 15 19:34:29.111796 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.111635    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b7d9cc0-9c8b-44c3-a848-fde069de28ac-must-gather-output\") pod \"must-gather-k5rrr\" (UID: \"9b7d9cc0-9c8b-44c3-a848-fde069de28ac\") " pod="openshift-must-gather-xxzwt/must-gather-k5rrr"
Sep 15 19:34:29.111943 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.111921    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b7d9cc0-9c8b-44c3-a848-fde069de28ac-must-gather-output\") pod \"must-gather-k5rrr\" (UID: \"9b7d9cc0-9c8b-44c3-a848-fde069de28ac\") " pod="openshift-must-gather-xxzwt/must-gather-k5rrr"
Sep 15 19:34:29.120187 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.120163    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r66tg\" (UniqueName: \"kubernetes.io/projected/9b7d9cc0-9c8b-44c3-a848-fde069de28ac-kube-api-access-r66tg\") pod \"must-gather-k5rrr\" (UID: \"9b7d9cc0-9c8b-44c3-a848-fde069de28ac\") " pod="openshift-must-gather-xxzwt/must-gather-k5rrr"
Sep 15 19:34:29.236551 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.236491    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxzwt/must-gather-k5rrr"
Sep 15 19:34:29.510725 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.510695    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxzwt/must-gather-k5rrr"]
Sep 15 19:34:29.512870 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:34:29.512842    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b7d9cc0_9c8b_44c3_a848_fde069de28ac.slice/crio-fce359f1cae0dcc165191bbf7b7b921b9afa9ae38aa8a6b5d66a9cd11d9b1ac9 WatchSource:0}: Error finding container fce359f1cae0dcc165191bbf7b7b921b9afa9ae38aa8a6b5d66a9cd11d9b1ac9: Status 404 returned error can't find the container with id fce359f1cae0dcc165191bbf7b7b921b9afa9ae38aa8a6b5d66a9cd11d9b1ac9
Sep 15 19:34:29.973958 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:29.973929    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/must-gather-k5rrr" event={"ID":"9b7d9cc0-9c8b-44c3-a848-fde069de28ac","Type":"ContainerStarted","Data":"fce359f1cae0dcc165191bbf7b7b921b9afa9ae38aa8a6b5d66a9cd11d9b1ac9"}
Sep 15 19:34:30.984074 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:30.983987    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/must-gather-k5rrr" event={"ID":"9b7d9cc0-9c8b-44c3-a848-fde069de28ac","Type":"ContainerStarted","Data":"f0ce14317b5786c2db74bbf643e9bc50a1f61289bc0148d319d327289e01492c"}
Sep 15 19:34:30.984074 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:30.984045    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/must-gather-k5rrr" event={"ID":"9b7d9cc0-9c8b-44c3-a848-fde069de28ac","Type":"ContainerStarted","Data":"4754d57c299aa74de27ad947c761e64216b6d186b19dd2642f2d24bee5ac284a"}
Sep 15 19:34:31.004970 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:31.004903    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xxzwt/must-gather-k5rrr" podStartSLOduration=2.136648337 podStartE2EDuration="3.00487914s" totalImagesPullingTime="868.230803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:34:31.001914488 +0000 UTC m=+2629.857767758" watchObservedRunningTime="2026-09-15 19:34:31.00487914 +0000 UTC m=+2629.860732410"
Sep 15 19:34:31.967898 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:31.967857    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5crfs_790ee8dd-f692-426e-be62-4eebe5bc293c/global-pull-secret-syncer/0.log"
Sep 15 19:34:32.182437 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:32.182407    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xnqp7_66b4cdae-b75b-4519-98d7-2dcb9b70fbf6/konnectivity-agent/0.log"
Sep 15 19:34:32.229625 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:32.229535    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-183.ec2.internal_78a56846a26782d79f6f1b96091c0b40/haproxy/0.log"
Sep 15 19:34:32.672867 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:32.672833    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-74969cc5f8-hnjvz" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:34:36.559085 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:36.559044    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d854f4d76-l8kht_6e1f175f-98f3-4c0b-8872-ea716c90656c/authorino/0.log"
Sep 15 19:34:38.244351 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.244321    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d041a31-b45b-4010-b40a-650daf542361/alertmanager/0.log"
Sep 15 19:34:38.265912 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.265885    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d041a31-b45b-4010-b40a-650daf542361/config-reloader/0.log"
Sep 15 19:34:38.290034 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.289963    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d041a31-b45b-4010-b40a-650daf542361/kube-rbac-proxy-web/0.log"
Sep 15 19:34:38.312788 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.312759    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d041a31-b45b-4010-b40a-650daf542361/kube-rbac-proxy/0.log"
Sep 15 19:34:38.335083 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.335055    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d041a31-b45b-4010-b40a-650daf542361/kube-rbac-proxy-metric/0.log"
Sep 15 19:34:38.358847 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.358817    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d041a31-b45b-4010-b40a-650daf542361/prom-label-proxy/0.log"
Sep 15 19:34:38.382113 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.382092    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6d041a31-b45b-4010-b40a-650daf542361/init-config-reloader/0.log"
Sep 15 19:34:38.430493 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.430467    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-58mhq_bd436ebb-24ed-4151-aa7c-c2baa894e4ff/cluster-monitoring-operator/0.log"
Sep 15 19:34:38.462264 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.462231    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lr5qw_d135cf38-92b1-465f-8353-6bf796fdebe9/kube-state-metrics/0.log"
Sep 15 19:34:38.487003 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.486943    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lr5qw_d135cf38-92b1-465f-8353-6bf796fdebe9/kube-rbac-proxy-main/0.log"
Sep 15 19:34:38.511252 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.511221    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lr5qw_d135cf38-92b1-465f-8353-6bf796fdebe9/kube-rbac-proxy-self/0.log"
Sep 15 19:34:38.537796 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.537773    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86f978c87-mbp62_2e5322c9-d822-452b-8cb2-7b115ac95f05/metrics-server/0.log"
Sep 15 19:34:38.572350 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.572320    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-j2hhw_9e848b73-2271-4a1e-a017-4f1a4d156f42/monitoring-plugin/0.log"
Sep 15 19:34:38.680045 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.680007    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8n6m_c2f1975f-6fc8-42e7-b359-ed338e9bfcb7/node-exporter/0.log"
Sep 15 19:34:38.699784 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.699759    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8n6m_c2f1975f-6fc8-42e7-b359-ed338e9bfcb7/kube-rbac-proxy/0.log"
Sep 15 19:34:38.720922 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.720893    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8n6m_c2f1975f-6fc8-42e7-b359-ed338e9bfcb7/init-textfile/0.log"
Sep 15 19:34:38.958303 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.958271    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3266dcc3-3962-43f6-a0e8-6661e668e406/prometheus/0.log"
Sep 15 19:34:38.985502 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:38.985471    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3266dcc3-3962-43f6-a0e8-6661e668e406/config-reloader/0.log"
Sep 15 19:34:39.008096 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:39.008066    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3266dcc3-3962-43f6-a0e8-6661e668e406/thanos-sidecar/0.log"
Sep 15 19:34:39.030865 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:39.030833    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3266dcc3-3962-43f6-a0e8-6661e668e406/kube-rbac-proxy-web/0.log"
Sep 15 19:34:39.053043 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:39.052926    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3266dcc3-3962-43f6-a0e8-6661e668e406/kube-rbac-proxy/0.log"
Sep 15 19:34:39.076086 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:39.076059    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3266dcc3-3962-43f6-a0e8-6661e668e406/kube-rbac-proxy-thanos/0.log"
Sep 15 19:34:39.100691 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:39.100360    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3266dcc3-3962-43f6-a0e8-6661e668e406/init-config-reloader/0.log"
Sep 15 19:34:40.995462 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:40.995428    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-p4frl_1fe06272-485c-452f-8eda-13dfb438a19a/networking-console-plugin/0.log"
Sep 15 19:34:41.686728 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:41.686692    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/1.log"
Sep 15 19:34:41.692674 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:41.692630    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-g4gm9_270f6947-813e-465b-9ad8-8428cf35b988/console-operator/2.log"
Sep 15 19:34:42.292318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.292286    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-qb526_d55b7ee3-bbf5-4dba-b743-964546e2bdeb/download-server/0.log"
Sep 15 19:34:42.292900 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.292346    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"]
Sep 15 19:34:42.299244 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.299217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.303972 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.303930    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"]
Sep 15 19:34:42.369827 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.369788    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-podres\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.370020 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.369845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-sys\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.370020 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.369905    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29qxb\" (UniqueName: \"kubernetes.io/projected/a5e70945-3d56-453e-a1b3-fb34669abc51-kube-api-access-29qxb\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.370020 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.369973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-proc\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.370020 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.369996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-lib-modules\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.470745 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.470696    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-podres\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.470934 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.470764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-sys\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.470934 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.470834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qxb\" (UniqueName: \"kubernetes.io/projected/a5e70945-3d56-453e-a1b3-fb34669abc51-kube-api-access-29qxb\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.470934 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.470877    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-podres\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.470934 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.470905    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-sys\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.471155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.470967    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-proc\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.471155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.470908    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-proc\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.471155 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.471012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-lib-modules\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.471296 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.471160    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5e70945-3d56-453e-a1b3-fb34669abc51-lib-modules\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.480662 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.480028    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qxb\" (UniqueName: \"kubernetes.io/projected/a5e70945-3d56-453e-a1b3-fb34669abc51-kube-api-access-29qxb\") pod \"perf-node-gather-daemonset-s69xx\" (UID: \"a5e70945-3d56-453e-a1b3-fb34669abc51\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.613340 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.613302    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:42.912998 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:42.912968    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"]
Sep 15 19:34:42.918090 ip-10-0-140-183 kubenswrapper[2556]: W0915 19:34:42.918045    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda5e70945_3d56_453e_a1b3_fb34669abc51.slice/crio-b07d80739e9902a934bb37165dfc5e406874a92913d2b7ce3e6194b1303ec8f9 WatchSource:0}: Error finding container b07d80739e9902a934bb37165dfc5e406874a92913d2b7ce3e6194b1303ec8f9: Status 404 returned error can't find the container with id b07d80739e9902a934bb37165dfc5e406874a92913d2b7ce3e6194b1303ec8f9
Sep 15 19:34:43.090258 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:43.089813    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx" event={"ID":"a5e70945-3d56-453e-a1b3-fb34669abc51","Type":"ContainerStarted","Data":"9ef2bc7a2c15b21a003c67bef620491155b72d1aaa5a7d276d28ad0f9d0b1bb4"}
Sep 15 19:34:43.090258 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:43.089879    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx" event={"ID":"a5e70945-3d56-453e-a1b3-fb34669abc51","Type":"ContainerStarted","Data":"b07d80739e9902a934bb37165dfc5e406874a92913d2b7ce3e6194b1303ec8f9"}
Sep 15 19:34:43.090258 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:43.090131    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:43.094358 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:43.094280    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-shdhj_3d82b329-5733-4457-b3c1-2171f98e9ff8/volume-data-source-validator/0.log"
Sep 15 19:34:43.108423 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:43.108346    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx" podStartSLOduration=1.108322335 podStartE2EDuration="1.108322335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 19:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:34:43.105567395 +0000 UTC m=+2641.961420655" watchObservedRunningTime="2026-09-15 19:34:43.108322335 +0000 UTC m=+2641.964175604"
Sep 15 19:34:44.437982 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:44.437950    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jf5l9_cdaba30b-50d4-422c-88d4-ac0f3dfb01c8/dns/0.log"
Sep 15 19:34:44.461135 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:44.461108    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jf5l9_cdaba30b-50d4-422c-88d4-ac0f3dfb01c8/kube-rbac-proxy/0.log"
Sep 15 19:34:44.520189 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:44.520154    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-64626_09057b3f-4b9b-4839-a4c4-f37bced3b634/dns-node-resolver/0.log"
Sep 15 19:34:45.165877 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:45.165835    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sx76p_275b46c7-18c0-4a76-8af3-1b6f17b2fbec/node-ca/0.log"
Sep 15 19:34:46.486101 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:46.486073    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65488ff6c7-8b8lm_42e7b5de-d52b-45b5-b510-f4bae1fd4fc8/kube-auth-proxy/0.log"
Sep 15 19:34:46.572265 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:46.572234    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-558d9b8847-jqkvg_3aa75d43-9279-4356-b3e5-0efaaa9aac59/payload-pre-processing/0.log"
Sep 15 19:34:46.597264 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:46.597232    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-74969cc5f8-hnjvz_3b536038-6bba-4a1a-a625-415d0e8712d6/payload-processing/0.log"
Sep 15 19:34:47.579136 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:47.579107    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5k94_300c7133-fd1a-4a29-95da-f0ebaaf366da/serve-healthcheck-canary/0.log"
Sep 15 19:34:47.673268 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:47.673227    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-558d9b8847-jqkvg" secret="" err="secret \"payload-processing-dockercfg-5z5d2\" not found"
Sep 15 19:34:48.046955 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:48.046919    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-8n8xn_f470421a-2714-4f71-9eb4-f2365287496a/insights-operator/1.log"
Sep 15 19:34:48.047351 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:48.047330    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-8n8xn_f470421a-2714-4f71-9eb4-f2365287496a/insights-operator/0.log"
Sep 15 19:34:48.068366 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:48.068330    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqshd_aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27/kube-rbac-proxy/0.log"
Sep 15 19:34:48.089635 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:48.089606    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqshd_aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27/exporter/0.log"
Sep 15 19:34:48.119148 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:48.119125    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqshd_aa5a7c3f-9f9a-46cd-808a-7857ae8d4d27/extractor/0.log"
Sep 15 19:34:49.110611 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:49.110575    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-s69xx"
Sep 15 19:34:50.368030 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:50.368006    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-747f7b9d6c-r9j59_0e0a67bb-859e-4cdf-aeee-5f1c35893446/manager/0.log"
Sep 15 19:34:50.685153 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:50.685116    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c9bcfd8cf-4j4zc_049f48e6-04a7-4f0a-b72c-ea1b0cdf86fd/manager/0.log"
Sep 15 19:34:50.943287 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:50.943202    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d9df7495-mqjpb_21361244-e888-41b9-8673-c3c5d86f9e39/manager/0.log"
Sep 15 19:34:50.960416 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:50.960388    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d9df7495-mqjpb_21361244-e888-41b9-8673-c3c5d86f9e39/copy-manifests/0.log"
Sep 15 19:34:52.516114 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:52.516084    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bcb5dcf99-6thb8_d25567c7-4a0f-4048-bee6-7f8bc83a599b/manager/0.log"
Sep 15 19:34:53.064848 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:53.064817    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65488ff6c7-8b8lm_42e7b5de-d52b-45b5-b510-f4bae1fd4fc8/kube-auth-proxy/0.log"
Sep 15 19:34:53.109351 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:53.109316    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-558d9b8847-jqkvg_3aa75d43-9279-4356-b3e5-0efaaa9aac59/payload-pre-processing/0.log"
Sep 15 19:34:53.122241 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:53.122219    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-74969cc5f8-hnjvz_3b536038-6bba-4a1a-a625-415d0e8712d6/payload-processing/0.log"
Sep 15 19:34:54.965109 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:54.965083    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65488ff6c7-8b8lm_42e7b5de-d52b-45b5-b510-f4bae1fd4fc8/kube-auth-proxy/0.log"
Sep 15 19:34:55.016540 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:55.016513    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-558d9b8847-jqkvg_3aa75d43-9279-4356-b3e5-0efaaa9aac59/payload-pre-processing/0.log"
Sep 15 19:34:55.032415 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:55.032378    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-74969cc5f8-hnjvz_3b536038-6bba-4a1a-a625-415d0e8712d6/payload-processing/0.log"
Sep 15 19:34:57.670314 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:57.670285    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hc2vq_d819910a-7306-41e9-ae7a-1eb057d41fad/migrator/0.log"
Sep 15 19:34:57.690901 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:57.690874    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hc2vq_d819910a-7306-41e9-ae7a-1eb057d41fad/graceful-termination/0.log"
Sep 15 19:34:59.194320 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.194288    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7mxn_4f43b24a-2748-44e9-848c-10b56daaf12b/kube-multus-additional-cni-plugins/0.log"
Sep 15 19:34:59.215052 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.215026    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7mxn_4f43b24a-2748-44e9-848c-10b56daaf12b/egress-router-binary-copy/0.log"
Sep 15 19:34:59.237187 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.237122    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7mxn_4f43b24a-2748-44e9-848c-10b56daaf12b/cni-plugins/0.log"
Sep 15 19:34:59.256011 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.255992    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7mxn_4f43b24a-2748-44e9-848c-10b56daaf12b/bond-cni-plugin/0.log"
Sep 15 19:34:59.277383 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.277365    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7mxn_4f43b24a-2748-44e9-848c-10b56daaf12b/routeoverride-cni/0.log"
Sep 15 19:34:59.299955 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.299935    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7mxn_4f43b24a-2748-44e9-848c-10b56daaf12b/whereabouts-cni-bincopy/0.log"
Sep 15 19:34:59.319744 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.319710    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7mxn_4f43b24a-2748-44e9-848c-10b56daaf12b/whereabouts-cni/0.log"
Sep 15 19:34:59.525318 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.525247    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hmgm4_b2d1e003-7ce3-4274-a974-58acb446076d/kube-multus/0.log"
Sep 15 19:34:59.596359 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.596327    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hfjb8_0724acd2-fe1b-4134-817d-a4fc7300ce5c/network-metrics-daemon/0.log"
Sep 15 19:34:59.613084 ip-10-0-140-183 kubenswrapper[2556]: I0915 19:34:59.613053    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hfjb8_0724acd2-fe1b-4134-817d-a4fc7300ce5c/kube-rbac-proxy/0.log"