Jul 30 20:21:01.407358 ip-10-0-140-21 systemd[1]: Starting Kubernetes Kubelet... Jul 30 20:21:01.852390 ip-10-0-140-21 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 20:21:01.852390 ip-10-0-140-21 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 20:21:01.852390 ip-10-0-140-21 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 20:21:01.852390 ip-10-0-140-21 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 20:21:01.852390 ip-10-0-140-21 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 20:21:01.853482 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.853403 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 20:21:01.858491 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858476 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:01.858491 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858491 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858495 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858498 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858502 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858505 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858508 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858517 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858521 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858524 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858527 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858530 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858532 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858535 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858537 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858540 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858543 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858545 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858548 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858550 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:01.858569 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858553 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858555 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858558 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858561 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858563 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858566 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858569 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858571 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858574 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858576 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858579 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858582 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858585 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858587 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858590 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858592 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858595 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858597 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858600 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:01.859024 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858602 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858605 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858608 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858610 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858613 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858615 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858617 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858620 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858622 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858625 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858627 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858630 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858632 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858637 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858641 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858644 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858649 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858652 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858655 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:01.859514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858658 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858662 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858666 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858669 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858672 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858675 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858677 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858680 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858683 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858686 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858689 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858691 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858694 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858696 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858699 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858702 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858705 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858707 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858710 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858712 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:01.860263 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858715 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858717 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858720 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858723 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858726 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858728 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858731 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858734 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858737 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858740 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858743 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858745 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858748 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858750 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858753 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858755 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858758 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858760 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858763 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858766 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:01.860794 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.858768 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861274 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861287 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861291 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861295 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861299 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861302 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861305 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861308 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861311 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861314 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861317 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861320 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861323 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861325 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861328 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:01.861324 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861331 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861334 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861337 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861340 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861342 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861345 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861348 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861350 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861353 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861355 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861358 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861360 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861363 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861366 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861368 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861371 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861378 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861382 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861386 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:01.861726 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861389 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861392 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861395 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861397 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861400 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861403 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861406 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861409 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861412 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861414 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861417 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861420 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861424 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861427 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861429 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861434 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861437 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861440 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861442 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861445 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:01.862297 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861448 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861451 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861453 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861456 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861459 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861461 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861464 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861466 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861470 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861473 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861476 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861479 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861481 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861484 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861487 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861489 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861492 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861495 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861497 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861500 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:01.862833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861503 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861505 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861508 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861510 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861513 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861515 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861519 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861523 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861526 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861529 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861533 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861537 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861540 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861543 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861546 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861549 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861552 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861554 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861558 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861561 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:01.863338 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861563 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861566 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861568 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861571 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.861574 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862028 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862034 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862038 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862041 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862044 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862046 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862049 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862067 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862070 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862072 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862075 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862078 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862080 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862083 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:01.863830 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862086 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862089 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862092 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862095 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862097 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862100 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862102 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862105 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862107 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862110 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862113 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862115 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862119 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862123 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862126 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862129 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862132 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862134 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862137 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862140 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:01.864319 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862143 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862146 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862150 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862153 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862156 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862159 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862162 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862165 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862169 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862171 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862174 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862177 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862179 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862182 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862185 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862188 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862191 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862193 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862196 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:01.864809 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862198 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862201 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862203 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862206 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862208 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862211 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862213 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862216 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862218 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862221 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862223 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862226 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862229 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862231 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862234 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862237 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862239 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862242 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862244 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862247 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:01.865286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862251 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862253 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862256 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862259 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862261 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862263 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862266 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862269 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862271 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862274 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862276 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862279 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862281 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862284 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862286 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862289 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862291 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862294 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862296 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862299 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:01.865778 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862301 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862304 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862307 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862309 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862312 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.862314 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862377 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862383 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862390 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862394 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862398 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862401 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862406 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862410 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862413 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862416 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862420 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862423 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862426 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862429 2575 flags.go:64] FLAG: --cgroup-root="" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862432 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862435 2575 flags.go:64] FLAG: --client-ca-file="" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862438 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 30 20:21:01.866286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862444 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862449 2575 flags.go:64] FLAG: --cluster-domain="" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862452 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862455 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862458 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862461 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862465 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862468 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862471 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862474 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862477 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862480 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862483 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862486 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862489 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862493 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862496 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862499 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862501 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862504 2575 flags.go:64] FLAG: --enable-server="true" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862507 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862513 2575 flags.go:64] FLAG: --event-burst="100" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862516 2575 flags.go:64] FLAG: --event-qps="50" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862519 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 20:21:01.866847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862522 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862525 2575 flags.go:64] FLAG: --eviction-hard="" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862529 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862531 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862534 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862537 2575 flags.go:64] FLAG: --eviction-soft="" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862540 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862543 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862548 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862551 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862554 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862557 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862560 2575 flags.go:64] FLAG: --feature-gates="" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862569 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862572 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862575 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862578 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862581 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862584 2575 flags.go:64] FLAG: --help="false" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862587 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-21.ec2.internal" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862590 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862593 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862595 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862599 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 20:21:01.867435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862602 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862605 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862608 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862611 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862614 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862618 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862621 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862624 2575 flags.go:64] FLAG: --kube-reserved="" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862626 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862629 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862632 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862635 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862638 2575 flags.go:64] FLAG: --lock-file="" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862641 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862644 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862647 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862653 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862656 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862659 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862662 2575 flags.go:64] FLAG: --logging-format="text" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862665 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862668 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862671 2575 flags.go:64] FLAG: --manifest-url="" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862674 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862678 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 20:21:01.868114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862681 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862685 2575 flags.go:64] FLAG: --max-pods="110" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862688 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862691 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862694 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862697 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862700 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862702 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862705 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862712 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862715 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862718 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862722 2575 flags.go:64] FLAG: --pod-cidr="" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862725 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862731 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862734 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862737 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862740 2575 flags.go:64] FLAG: --port="10250" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862743 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862746 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-063e04637770c6bed" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862749 2575 flags.go:64] FLAG: --qos-reserved="" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862752 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862755 2575 flags.go:64] FLAG: --register-node="true" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862759 2575 flags.go:64] FLAG: --register-with-taints="" Jul 30 20:21:01.868759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862763 2575 flags.go:64] FLAG: --registry-burst="10" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862766 2575 flags.go:64] FLAG: --registry-qps="5" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862769 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862772 2575 flags.go:64] FLAG: --reserved-memory="" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862776 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862779 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862781 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862784 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862787 2575 flags.go:64] FLAG: --runonce="false" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862790 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862793 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862796 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862799 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862802 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862805 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862808 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862811 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862814 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862817 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862820 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862823 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862828 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862830 2575 flags.go:64] FLAG: --system-cgroups="" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862833 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862839 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 20:21:01.869376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862842 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862845 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862850 2575 flags.go:64] FLAG: --tls-min-version="" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862852 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862855 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862858 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862865 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862868 2575 flags.go:64] FLAG: --v="2" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862872 2575 flags.go:64] FLAG: --version="false" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862876 2575 flags.go:64] FLAG: --vmodule="" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862880 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.862883 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863001 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863005 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863008 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863010 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863013 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863016 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863018 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863021 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863024 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863026 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:01.869982 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863029 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863032 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863035 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863037 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863040 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863042 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863077 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863081 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863084 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863087 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863090 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863093 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863095 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863098 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863101 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863103 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863107 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863109 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863112 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863115 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:01.870546 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863118 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863121 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863123 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863126 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863129 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863131 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863134 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863136 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863139 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863141 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863145 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863149 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863152 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863155 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863158 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863160 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863163 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863166 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863170 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:01.871063 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863173 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863177 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863181 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863183 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863186 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863189 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863191 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863194 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863196 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863200 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863203 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863206 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863208 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863211 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863214 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863217 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863219 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863222 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863224 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:01.871547 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863227 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863229 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863232 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863234 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863237 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863239 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863243 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863247 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863250 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863252 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863255 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863257 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863261 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863264 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863267 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863269 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863272 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863274 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863277 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863280 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:01.872035 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863282 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863285 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863289 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863291 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863294 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863296 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863299 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863301 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863306 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863308 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.863311 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.863315 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.870008 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.870022 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.870041 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 20:21:01.873427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.870046 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870116 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870121 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870124 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870127 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870130 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870132 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870135 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870137 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870140 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870143 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870145 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870148 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870150 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870153 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870155 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870158 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870160 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870163 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870166 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:01.873811 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870169 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870171 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870174 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870176 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870178 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870181 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870184 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870186 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870189 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870192 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870196 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870199 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870201 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870204 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870206 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870209 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870211 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870214 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870216 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870219 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:01.874313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870222 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870224 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870227 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870231 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870234 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870237 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870240 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870243 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870245 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870248 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870251 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870254 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870256 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870259 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870262 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870265 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870267 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870270 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870272 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:01.874816 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870275 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870277 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870280 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870283 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870287 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870290 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870293 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870295 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870298 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870300 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870303 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870307 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870311 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870314 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870317 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870319 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870322 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870324 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870327 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:01.875307 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870329 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870332 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870334 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870337 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870340 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870342 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870345 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870347 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870349 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870352 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870354 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870357 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870359 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870362 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870366 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870369 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870372 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870375 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870378 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870382 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:01.875769 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870384 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870387 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.870392 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870494 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870499 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870502 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870505 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870508 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870510 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870513 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870516 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870518 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870520 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870523 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870525 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:01.876330 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870528 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870532 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870535 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870538 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870540 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870542 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870545 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870547 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870550 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870553 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870555 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870558 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870561 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870563 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870566 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870569 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870572 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870574 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870577 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870579 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:01.876694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870582 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870584 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870587 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870589 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870592 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870594 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870596 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870599 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870601 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870604 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870606 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870609 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870611 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870614 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870616 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870619 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870621 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870624 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870626 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870629 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:01.877209 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870631 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870634 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870636 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870639 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870641 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870644 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870647 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870649 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870652 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870655 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870657 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870660 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870662 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870665 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870667 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870669 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870673 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870676 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870680 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:01.877698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870682 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870685 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870688 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870690 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870694 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870697 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870700 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870703 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870707 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870709 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870712 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870715 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870717 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870720 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870723 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870726 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870729 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870731 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870734 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870737 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:01.878183 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870739 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870742 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870744 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870747 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870750 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870752 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870755 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:01.870757 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.870762 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.870868 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.873142 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.874820 2575 server.go:1019] "Starting client certificate rotation" Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.874921 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 20:21:01.878694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.874949 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 20:21:01.879023 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.878406 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 20:21:01.879490 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.879474 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 20:21:01.893461 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.893440 2575 log.go:25] "Validated CRI v1 runtime API" Jul 30 20:21:01.895868 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.895852 2575 log.go:25] "Validated CRI v1 image API" Jul 30 20:21:01.896624 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.896611 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 20:21:01.899109 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.899090 2575 fs.go:135] Filesystem UUIDs: map[77ca8a33-d62c-42b1-9f7e-51002409a231:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bc037209-75fb-44b5-bd4d-69351e36a55b:/dev/nvme0n1p3] Jul 30 20:21:01.899185 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.899106 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 20:21:01.904525 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.904505 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 20:21:01.905018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.904915 2575 manager.go:216] Machine: {Timestamp:2026-07-30 20:21:01.902622148 +0000 UTC m=+0.367760487 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097076 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214df95083c3364dd1b8f35f49be3c SystemUUID:ec214df9-5083-c336-4dd1-b8f35f49be3c BootID:1966899d-4498-4f07-ab57-46bb14fe03f4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:67:0c:fb:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:67:0c:fb:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:9a:10:83:26:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 20:21:01.905018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.905017 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 20:21:01.905139 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.905106 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 20:21:01.908082 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.908041 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 20:21:01.908254 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.908087 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 20:21:01.908985 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.908973 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 20:21:01.909022 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.908989 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 20:21:01.909022 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.909000 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 20:21:01.909022 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.909018 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 20:21:01.910317 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.910305 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 20:21:01.915643 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.915630 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 30 20:21:01.915784 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.915773 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 20:21:01.915827 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.915811 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 30 20:21:01.915827 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.915824 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 20:21:01.915894 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.915842 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 20:21:01.915894 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.915857 2575 kubelet.go:398] "Adding apiserver pod source" Jul 30 20:21:01.915894 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.915878 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 20:21:01.916300 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.916286 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 20:21:01.916350 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.916311 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 20:21:01.918768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.918754 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 20:21:01.920673 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.920660 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 20:21:01.920717 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.920701 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 20:21:01.922267 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922255 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 20:21:01.922331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922271 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 20:21:01.922331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922277 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 20:21:01.922331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922283 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 20:21:01.922331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922294 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 20:21:01.922331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922302 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 20:21:01.922331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922313 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 20:21:01.922331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922321 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 20:21:01.922331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922330 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 20:21:01.922536 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922336 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 20:21:01.922536 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922344 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 20:21:01.922536 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.922350 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 20:21:01.924092 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.924079 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 20:21:01.924132 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.924094 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 20:21:01.926194 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.926166 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 20:21:01.926283 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.926267 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 20:21:01.927520 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.927505 2575 server.go:1268] "Started kubelet" Jul 30 20:21:01.927628 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.927573 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 20:21:01.927678 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.927639 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 20:21:01.927708 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.927700 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 20:21:01.929123 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.929106 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 20:21:01.929457 ip-10-0-140-21 systemd[1]: Started Kubernetes Kubelet. Jul 30 20:21:01.934134 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.934115 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 30 20:21:01.934223 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.934142 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 20:21:01.939941 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.939919 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 20:21:01.939941 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.939922 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 20:21:01.940179 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.940159 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-21.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 20:21:01.940235 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.940186 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:01.940876 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.940859 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 20:21:01.940876 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.940859 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 20:21:01.941020 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.940886 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 20:21:01.941020 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.940986 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 30 20:21:01.941020 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.940994 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 30 20:21:01.941177 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.941099 2575 factory.go:55] Registering systemd factory Jul 30 20:21:01.941177 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.941114 2575 factory.go:223] Registration of the systemd container factory successfully Jul 30 20:21:01.941310 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.941295 2575 factory.go:153] Registering CRI-O factory Jul 30 20:21:01.941310 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.941311 2575 factory.go:223] Registration of the crio container factory successfully Jul 30 20:21:01.941408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.941357 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 20:21:01.941408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.941376 2575 factory.go:103] Registering Raw factory Jul 30 20:21:01.941408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.941386 2575 manager.go:1196] Started watching for new ooms in manager Jul 30 20:21:01.941661 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.940387 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-21.ec2.internal.18c72ad2bfeeeb1a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-21.ec2.internal,UID:ip-10-0-140-21.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-21.ec2.internal,},FirstTimestamp:2026-07-30 20:21:01.92729577 +0000 UTC m=+0.392434109,LastTimestamp:2026-07-30 20:21:01.92729577 +0000 UTC m=+0.392434109,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-21.ec2.internal,}" Jul 30 20:21:01.941780 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.941733 2575 manager.go:349] Starting recovery of all containers Jul 30 20:21:01.946258 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.946231 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 20:21:01.946258 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.946247 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 20:21:01.953234 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.953212 2575 manager.go:354] Recovery completed Jul 30 20:21:01.957445 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.957433 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:01.959154 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.959137 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:01.959241 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.959164 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:01.959241 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.959182 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:01.959726 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.959710 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 20:21:01.959726 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.959724 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 20:21:01.959848 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.959745 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 30 20:21:01.959848 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.959778 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:01.960521 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.960508 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:01.960574 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.960533 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:01.960574 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.960546 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:01.961399 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.961317 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-21.ec2.internal.18c72ad2c1d504d2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-21.ec2.internal,UID:ip-10-0-140-21.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-21.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-21.ec2.internal,},FirstTimestamp:2026-07-30 20:21:01.95915285 +0000 UTC m=+0.424291192,LastTimestamp:2026-07-30 20:21:01.95915285 +0000 UTC m=+0.424291192,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-21.ec2.internal,}" Jul 30 20:21:01.961927 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.961915 2575 policy_none.go:49] "None policy: Start" Jul 30 20:21:01.961971 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.961931 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 20:21:01.961971 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.961941 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 20:21:01.964089 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.964072 2575 policy_none.go:47] "Start" Jul 30 20:21:01.964172 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.964153 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lg8lj" Jul 30 20:21:01.972065 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:01.972037 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lg8lj" Jul 30 20:21:01.972462 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:01.972388 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-21.ec2.internal.18c72ad2c1d544aa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-21.ec2.internal,UID:ip-10-0-140-21.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-21.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-21.ec2.internal,},FirstTimestamp:2026-07-30 20:21:01.959169194 +0000 UTC m=+0.424307532,LastTimestamp:2026-07-30 20:21:01.959169194 +0000 UTC m=+0.424307532,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-21.ec2.internal,}" Jul 30 20:21:02.005463 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.005427 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 20:21:02.017270 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.006561 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 20:21:02.017270 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.006592 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 20:21:02.017270 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.006608 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 20:21:02.017270 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.006637 2575 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 30 20:21:02.017270 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.008727 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:02.040672 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.040653 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.100674 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.100524 2575 manager.go:337] "Starting Device Plugin manager" Jul 30 20:21:02.100758 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.100726 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 20:21:02.100758 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.100740 2575 server.go:87] "Starting device plugin registration server" Jul 30 20:21:02.100978 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.100967 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 20:21:02.101011 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.100982 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 20:21:02.101127 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.101104 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 20:21:02.101256 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.101203 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 20:21:02.101256 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.101214 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 20:21:02.101720 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.101701 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 20:21:02.101793 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.101743 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.106725 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.106681 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal"] Jul 30 20:21:02.106783 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.106740 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:02.107504 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.107488 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:02.107630 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.107519 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:02.107630 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.107558 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:02.109761 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.109747 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:02.109867 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.109853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.109916 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.109885 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:02.110452 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.110438 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:02.110529 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.110467 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:02.110529 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.110483 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:02.110529 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.110441 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:02.110680 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.110540 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:02.110680 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.110550 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:02.112474 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.112459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.112550 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.112486 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:02.113248 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.113233 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:02.113316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.113260 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:02.113316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.113270 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:02.131554 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.131534 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-21.ec2.internal\" not found" node="ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.135839 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.135821 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-21.ec2.internal\" not found" node="ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.142662 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.142646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8c674c09deef1f48c2b8c515f7e6e86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal\" (UID: \"e8c674c09deef1f48c2b8c515f7e6e86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.142744 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.142668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c79bb9ccdd4a990748877883d79006b3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-21.ec2.internal\" (UID: \"c79bb9ccdd4a990748877883d79006b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.142744 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.142686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8c674c09deef1f48c2b8c515f7e6e86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal\" (UID: \"e8c674c09deef1f48c2b8c515f7e6e86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.151421 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.151402 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-21.ec2.internal\" not found" node="ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.201910 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.201895 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:02.202560 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.202542 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:02.202640 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.202569 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:02.202640 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.202579 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:02.202640 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.202599 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.212207 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.212189 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.212207 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.212209 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-21.ec2.internal\": node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.234958 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.234937 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.243188 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.243167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8c674c09deef1f48c2b8c515f7e6e86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal\" (UID: \"e8c674c09deef1f48c2b8c515f7e6e86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.243261 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.243195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8c674c09deef1f48c2b8c515f7e6e86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal\" (UID: \"e8c674c09deef1f48c2b8c515f7e6e86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.243261 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.243218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c79bb9ccdd4a990748877883d79006b3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-21.ec2.internal\" (UID: \"c79bb9ccdd4a990748877883d79006b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.243338 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.243269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8c674c09deef1f48c2b8c515f7e6e86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal\" (UID: \"e8c674c09deef1f48c2b8c515f7e6e86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.243338 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.243314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8c674c09deef1f48c2b8c515f7e6e86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal\" (UID: \"e8c674c09deef1f48c2b8c515f7e6e86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.243410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.243340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c79bb9ccdd4a990748877883d79006b3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-21.ec2.internal\" (UID: \"c79bb9ccdd4a990748877883d79006b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.335365 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.335341 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.435194 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.435130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.436203 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.436185 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.439572 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.439555 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" Jul 30 20:21:02.536423 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.536385 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.637719 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.637677 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.738239 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.738133 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.842174 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.842125 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.875718 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.875685 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 20:21:02.876256 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.875894 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 20:21:02.942322 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:02.942285 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:02.977681 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.977645 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 20:16:02 +0000 UTC" deadline="2028-04-05 20:54:35.059266039 +0000 UTC" Jul 30 20:21:02.977843 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:02.977692 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14760h33m32.081594166s" Jul 30 20:21:03.042151 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.042079 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:03.042356 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:03.042338 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:03.090972 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.090936 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:03.133558 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.133513 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:03.134635 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.134619 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:03.134694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.134656 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:03.134694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.134670 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:03.143089 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:03.143052 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:03.164714 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:03.164664 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79bb9ccdd4a990748877883d79006b3.slice/crio-caf7563e52dc980ace9242c61fa2432227c049dbd95dd79ef918c195342ecb59 WatchSource:0}: Error finding container caf7563e52dc980ace9242c61fa2432227c049dbd95dd79ef918c195342ecb59: Status 404 returned error can't find the container with id caf7563e52dc980ace9242c61fa2432227c049dbd95dd79ef918c195342ecb59 Jul 30 20:21:03.165270 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:03.165248 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c674c09deef1f48c2b8c515f7e6e86.slice/crio-aed69f04795f6165bba76bd8f820ad4322da9228266fbf8bf8d50c27e7575044 WatchSource:0}: Error finding container aed69f04795f6165bba76bd8f820ad4322da9228266fbf8bf8d50c27e7575044: Status 404 returned error can't find the container with id aed69f04795f6165bba76bd8f820ad4322da9228266fbf8bf8d50c27e7575044 Jul 30 20:21:03.170021 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.170006 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 20:21:03.244840 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:03.244791 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:03.345859 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:03.345769 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-21.ec2.internal\" not found" Jul 30 20:21:03.421599 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.421562 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:03.441265 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.441232 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" Jul 30 20:21:03.459470 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.459445 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 20:21:03.460847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.460829 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" Jul 30 20:21:03.479120 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.479093 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 20:21:03.916856 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.916740 2575 apiserver.go:52] "Watching apiserver" Jul 30 20:21:03.930342 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.929309 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 20:21:03.930342 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.929875 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zkqt8","openshift-ovn-kubernetes/ovnkube-node-f99c7","kube-system/konnectivity-agent-st6vb","openshift-cluster-node-tuning-operator/tuned-vxfr2","openshift-dns/node-resolver-5mfzp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal","openshift-multus/multus-l85qb","openshift-multus/network-metrics-daemon-dgsbq","openshift-network-diagnostics/network-check-target-k8266","openshift-network-operator/iptables-alerter-9m2fz","kube-system/global-pull-secret-syncer-6htjd","kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5","openshift-image-registry/node-ca-ktn4b"] Jul 30 20:21:03.933200 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.932956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.935491 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.935457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:03.936054 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.936030 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 20:21:03.936440 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.936421 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8rqbh\"" Jul 30 20:21:03.936628 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.936610 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 20:21:03.936854 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.936825 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 20:21:03.937266 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.937248 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 20:21:03.938020 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.937834 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 20:21:03.939579 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.939559 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8npwj\"" Jul 30 20:21:03.939684 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.939577 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 20:21:03.939968 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.939905 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 20:21:03.941723 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.941675 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 20:21:03.942307 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.942288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:03.943612 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.943515 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.944921 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.944903 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 20:21:03.945204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.945186 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-955jf\"" Jul 30 20:21:03.945286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.945250 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 20:21:03.946286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.946266 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:03.947096 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.947068 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:03.947317 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.947295 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pwxlr\"" Jul 30 20:21:03.947539 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.947517 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:03.948547 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.948529 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 20:21:03.948848 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.948832 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 20:21:03.948926 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.948908 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fq9fk\"" Jul 30 20:21:03.949004 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.948833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:03.951446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.951429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:03.951542 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:03.951507 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:03.951895 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.951875 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zslzk\"" Jul 30 20:21:03.952027 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.951982 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 20:21:03.952144 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.952116 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 20:21:03.952220 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.951881 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 20:21:03.953180 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.953158 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 20:21:03.953545 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.953515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-cni-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.953632 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.953551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-socket-dir-parent\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.953632 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.953579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-cni-bin\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.953632 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.953609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-cni-multus\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.953788 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.953644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-daemon-config\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.953788 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.953670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-socket-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.953694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ede1ef48-7bac-40e6-a395-7357ac99d373-agent-certs\") pod \"konnectivity-agent-st6vb\" (UID: \"ede1ef48-7bac-40e6-a395-7357ac99d373\") " pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.954792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4343dc18-e764-4840-a67e-fa8389bf80eb-cni-binary-copy\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.954865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-netns\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.954894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-device-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.954920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.954963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-modprobe-d\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.954990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysconfig\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.955017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-kubernetes\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.955050 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.955044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysctl-d\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.955456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-lib-modules\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.955498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-tuned\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.955586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.955550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/364bec14-8852-434b-a5d4-6d1bc081ad1d-tmp\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.956283 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6vv\" (UniqueName: \"kubernetes.io/projected/364bec14-8852-434b-a5d4-6d1bc081ad1d-kube-api-access-cw6vv\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.956283 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-kubelet\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.956283 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:03.956445 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-sys\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.956445 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-var-lib-kubelet\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.956445 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-host\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.956594 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-cnibin\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.956594 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5sp\" (UniqueName: \"kubernetes.io/projected/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-kube-api-access-cr5sp\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:03.956594 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-sys-fs\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:03.956754 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysctl-conf\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.956754 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-systemd\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.956754 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-run\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:03.956754 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-system-cni-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.956754 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956713 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-multus-certs\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.957058 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfjf\" (UniqueName: \"kubernetes.io/projected/4343dc18-e764-4840-a67e-fa8389bf80eb-kube-api-access-nbfjf\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.957058 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ede1ef48-7bac-40e6-a395-7357ac99d373-konnectivity-ca\") pod \"konnectivity-agent-st6vb\" (UID: \"ede1ef48-7bac-40e6-a395-7357ac99d373\") " pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:03.957058 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-k8s-cni-cncf-io\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.957058 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-etc-kubernetes\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.957058 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-os-release\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.957058 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-hostroot\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.957058 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-conf-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:03.957058 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.956987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-registration-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:03.957552 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.957534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:03.957658 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.957619 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 20:21:03.958427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.958408 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 20:21:03.958508 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.958457 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-626sr\"" Jul 30 20:21:03.960181 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.960156 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 20:21:03.960686 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.960667 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 20:21:03.960795 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.960723 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 20:21:03.960864 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.960797 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 20:21:03.961074 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.961038 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52d8c\"" Jul 30 20:21:03.962122 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.962053 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 20:21:03.962122 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.962104 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 20:21:03.962493 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.962475 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:03.964966 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.964946 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:03.965734 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.965718 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2pxpq\"" Jul 30 20:21:03.965942 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.965926 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 20:21:03.966164 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.966144 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:03.966235 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.966149 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:03.966289 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:03.966264 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:03.969947 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.969926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:03.970194 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:03.970168 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:03.980345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.980318 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxk2s" Jul 30 20:21:03.986147 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:03.986116 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxk2s" Jul 30 20:21:04.013399 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.013331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" event={"ID":"c79bb9ccdd4a990748877883d79006b3","Type":"ContainerStarted","Data":"caf7563e52dc980ace9242c61fa2432227c049dbd95dd79ef918c195342ecb59"} Jul 30 20:21:04.018446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.018375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" event={"ID":"e8c674c09deef1f48c2b8c515f7e6e86","Type":"ContainerStarted","Data":"aed69f04795f6165bba76bd8f820ad4322da9228266fbf8bf8d50c27e7575044"} Jul 30 20:21:04.042627 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.042589 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.057793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-hostroot\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.057823 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.057862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-conf-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.057883 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-hostroot\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.057900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr69s\" (UniqueName: \"kubernetes.io/projected/dfc8c488-8fee-4f10-a041-8d67bc0569a8-kube-api-access-kr69s\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.057975 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-conf-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.057976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-socket-dir-parent\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-cni-bin\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-daemon-config\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-socket-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058112 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-cni-bin\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-modprobe-d\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058137 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-socket-dir-parent\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058162 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-os-release\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4343dc18-e764-4840-a67e-fa8389bf80eb-cni-binary-copy\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-netns\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-kubernetes\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.059356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-modprobe-d\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058348 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-socket-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-var-lib-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058352 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-kubernetes\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-netns\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-etc-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-lib-modules\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/364bec14-8852-434b-a5d4-6d1bc081ad1d-tmp\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6vv\" (UniqueName: \"kubernetes.io/projected/364bec14-8852-434b-a5d4-6d1bc081ad1d-kube-api-access-cw6vv\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-system-cni-dir\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-log-socket\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed08e791-3ce9-4d63-8283-489d4fd655e6-iptables-alerter-script\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-kubelet\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.060613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-sys\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-systemd-units\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-cni-bin\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdsg\" (UniqueName: \"kubernetes.io/projected/2b4cea60-f228-49bb-b0d7-5122d386a695-kube-api-access-lxdsg\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058913 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-daemon-config\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-systemd\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.058959 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhq2c\" (UniqueName: \"kubernetes.io/projected/da966ad2-6153-4de3-8d61-4f80ce2fda88-kube-api-access-dhq2c\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtvg\" (UniqueName: \"kubernetes.io/projected/ff93e08a-42e5-457b-96c2-a54192e5518b-kube-api-access-sqtvg\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059029 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-run-netns\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-systemd\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059170 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4343dc18-e764-4840-a67e-fa8389bf80eb-cni-binary-copy\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-multus-certs\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-kubelet\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-kubelet\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-slash\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.061492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-cni-netd\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059357 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-sys\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-systemd\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-k8s-cni-cncf-io\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-etc-kubernetes\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da966ad2-6153-4de3-8d61-4f80ce2fda88-host\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-ovnkube-script-lib\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-registration-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-cni-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-cni-multus\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ede1ef48-7bac-40e6-a395-7357ac99d373-agent-certs\") pod \"konnectivity-agent-st6vb\" (UID: \"ede1ef48-7bac-40e6-a395-7357ac99d373\") " pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysconfig\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysctl-d\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-var-lib-kubelet\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.059925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-host\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4cea60-f228-49bb-b0d7-5122d386a695-ovn-node-metrics-cert\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.062443 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-device-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff93e08a-42e5-457b-96c2-a54192e5518b-tmp-dir\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-tuned\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff93e08a-42e5-457b-96c2-a54192e5518b-hosts-file\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060288 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19e1b2f5-58e3-423b-80d7-51670c463768-kubelet-config\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19e1b2f5-58e3-423b-80d7-51670c463768-dbus\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-run-ovn-kubernetes\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-cnibin\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5sp\" (UniqueName: \"kubernetes.io/projected/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-kube-api-access-cr5sp\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-sys-fs\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysctl-conf\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-run\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da966ad2-6153-4de3-8d61-4f80ce2fda88-serviceca\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060611 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-cnibin\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.063428 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-system-cni-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060796 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysctl-d\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfjf\" (UniqueName: \"kubernetes.io/projected/4343dc18-e764-4840-a67e-fa8389bf80eb-kube-api-access-nbfjf\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ede1ef48-7bac-40e6-a395-7357ac99d373-konnectivity-ca\") pod \"konnectivity-agent-st6vb\" (UID: \"ede1ef48-7bac-40e6-a395-7357ac99d373\") " pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-multus-certs\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.060983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-ovn\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-node-log\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061030 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-lib-modules\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-ovnkube-config\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-run\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed08e791-3ce9-4d63-8283-489d4fd655e6-host-slash\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-system-cni-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5km7\" (UniqueName: \"kubernetes.io/projected/41464378-d96a-4890-bcdb-fb23717b7593-kube-api-access-l5km7\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061195 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-var-lib-cni-multus\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.064316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-env-overrides\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061217 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-var-lib-kubelet\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stf9c\" (UniqueName: \"kubernetes.io/projected/ed08e791-3ce9-4d63-8283-489d4fd655e6-kube-api-access-stf9c\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-registration-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-os-release\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-host\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-host-run-k8s-cni-cncf-io\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-device-dir\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061398 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-os-release\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-etc-kubernetes\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysconfig\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061508 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-sys-fs\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061546 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-multus-cni-dir\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.061629 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-sysctl-conf\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.062029 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ede1ef48-7bac-40e6-a395-7357ac99d373-konnectivity-ca\") pod \"konnectivity-agent-st6vb\" (UID: \"ede1ef48-7bac-40e6-a395-7357ac99d373\") " pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.062767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4343dc18-e764-4840-a67e-fa8389bf80eb-cnibin\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.064283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/364bec14-8852-434b-a5d4-6d1bc081ad1d-tmp\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.065345 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.064718 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ede1ef48-7bac-40e6-a395-7357ac99d373-agent-certs\") pod \"konnectivity-agent-st6vb\" (UID: \"ede1ef48-7bac-40e6-a395-7357ac99d373\") " pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:04.068323 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.068295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6vv\" (UniqueName: \"kubernetes.io/projected/364bec14-8852-434b-a5d4-6d1bc081ad1d-kube-api-access-cw6vv\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.069158 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.069087 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/364bec14-8852-434b-a5d4-6d1bc081ad1d-etc-tuned\") pod \"tuned-vxfr2\" (UID: \"364bec14-8852-434b-a5d4-6d1bc081ad1d\") " pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.069770 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.069747 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfjf\" (UniqueName: \"kubernetes.io/projected/4343dc18-e764-4840-a67e-fa8389bf80eb-kube-api-access-nbfjf\") pod \"multus-l85qb\" (UID: \"4343dc18-e764-4840-a67e-fa8389bf80eb\") " pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.071435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.071411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5sp\" (UniqueName: \"kubernetes.io/projected/bda0552e-d5f2-4f26-9feb-5069fdbd32ff-kube-api-access-cr5sp\") pod \"aws-ebs-csi-driver-node-6vpd5\" (UID: \"bda0552e-d5f2-4f26-9feb-5069fdbd32ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.162427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr69s\" (UniqueName: \"kubernetes.io/projected/dfc8c488-8fee-4f10-a041-8d67bc0569a8-kube-api-access-kr69s\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:04.162427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-os-release\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-var-lib-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-os-release\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-etc-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-etc-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-var-lib-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-system-cni-dir\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-system-cni-dir\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.162682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-log-socket\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.163251 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed08e791-3ce9-4d63-8283-489d4fd655e6-iptables-alerter-script\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.163251 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-systemd-units\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.163251 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-cni-bin\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.163251 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.162813 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:04.163251 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdsg\" (UniqueName: \"kubernetes.io/projected/2b4cea60-f228-49bb-b0d7-5122d386a695-kube-api-access-lxdsg\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.163251 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhq2c\" (UniqueName: \"kubernetes.io/projected/da966ad2-6153-4de3-8d61-4f80ce2fda88-kube-api-access-dhq2c\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.163543 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.162878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-systemd-units\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.163594 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.163573 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs podName:dfc8c488-8fee-4f10-a041-8d67bc0569a8 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:04.662878045 +0000 UTC m=+3.128016375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs") pod "network-metrics-daemon-dgsbq" (UID: "dfc8c488-8fee-4f10-a041-8d67bc0569a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:04.163872 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.163848 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.163872 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.163859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.163991 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.163891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtvg\" (UniqueName: \"kubernetes.io/projected/ff93e08a-42e5-457b-96c2-a54192e5518b-kube-api-access-sqtvg\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.163991 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.163961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-cni-bin\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed08e791-3ce9-4d63-8283-489d4fd655e6-iptables-alerter-script\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-run-netns\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-systemd\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-kubelet\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-slash\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-systemd\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-cni-netd\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da966ad2-6153-4de3-8d61-4f80ce2fda88-host\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-openvswitch\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164351 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-run-netns\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-ovnkube-script-lib\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4cea60-f228-49bb-b0d7-5122d386a695-ovn-node-metrics-cert\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164636 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-cni-netd\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff93e08a-42e5-457b-96c2-a54192e5518b-tmp-dir\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.165967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff93e08a-42e5-457b-96c2-a54192e5518b-hosts-file\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19e1b2f5-58e3-423b-80d7-51670c463768-kubelet-config\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19e1b2f5-58e3-423b-80d7-51670c463768-dbus\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da966ad2-6153-4de3-8d61-4f80ce2fda88-host\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164895 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-kubelet\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-run-ovn-kubernetes\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da966ad2-6153-4de3-8d61-4f80ce2fda88-serviceca\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff93e08a-42e5-457b-96c2-a54192e5518b-hosts-file\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.164976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-slash\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165103 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-run-ovn-kubernetes\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-cnibin\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165188 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19e1b2f5-58e3-423b-80d7-51670c463768-kubelet-config\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19e1b2f5-58e3-423b-80d7-51670c463768-dbus\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff93e08a-42e5-457b-96c2-a54192e5518b-tmp-dir\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-ovn\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-node-log\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.166967 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-ovnkube-config\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.165402 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41464378-d96a-4890-bcdb-fb23717b7593-cnibin\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed08e791-3ce9-4d63-8283-489d4fd655e6-host-slash\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.165462 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret podName:19e1b2f5-58e3-423b-80d7-51670c463768 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:04.665442759 +0000 UTC m=+3.130581092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret") pod "global-pull-secret-syncer-6htjd" (UID: "19e1b2f5-58e3-423b-80d7-51670c463768") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165491 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-node-log\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed08e791-3ce9-4d63-8283-489d4fd655e6-host-slash\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-run-ovn\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5km7\" (UniqueName: \"kubernetes.io/projected/41464378-d96a-4890-bcdb-fb23717b7593-kube-api-access-l5km7\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-env-overrides\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stf9c\" (UniqueName: \"kubernetes.io/projected/ed08e791-3ce9-4d63-8283-489d4fd655e6-kube-api-access-stf9c\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165726 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-ovnkube-script-lib\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.165886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da966ad2-6153-4de3-8d61-4f80ce2fda88-serviceca\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.167851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.166082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b4cea60-f228-49bb-b0d7-5122d386a695-log-socket\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.168713 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.166238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-env-overrides\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.168713 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.166681 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41464378-d96a-4890-bcdb-fb23717b7593-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.168713 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.166932 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b4cea60-f228-49bb-b0d7-5122d386a695-ovnkube-config\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.170537 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.170514 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4cea60-f228-49bb-b0d7-5122d386a695-ovn-node-metrics-cert\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.179356 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.179184 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:04.179356 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.179214 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:04.179356 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.179231 2575 projected.go:196] Error preparing data for projected volume kube-api-access-79c4s for pod openshift-network-diagnostics/network-check-target-k8266: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:04.179356 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.179324 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s podName:27a0cc53-05f6-454e-9feb-28dd6673da84 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:04.679301544 +0000 UTC m=+3.144439897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-79c4s" (UniqueName: "kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s") pod "network-check-target-k8266" (UID: "27a0cc53-05f6-454e-9feb-28dd6673da84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:04.179990 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.179952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhq2c\" (UniqueName: \"kubernetes.io/projected/da966ad2-6153-4de3-8d61-4f80ce2fda88-kube-api-access-dhq2c\") pod \"node-ca-ktn4b\" (UID: \"da966ad2-6153-4de3-8d61-4f80ce2fda88\") " pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.180644 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.180610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr69s\" (UniqueName: \"kubernetes.io/projected/dfc8c488-8fee-4f10-a041-8d67bc0569a8-kube-api-access-kr69s\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:04.181393 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.181370 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5km7\" (UniqueName: \"kubernetes.io/projected/41464378-d96a-4890-bcdb-fb23717b7593-kube-api-access-l5km7\") pod \"multus-additional-cni-plugins-zkqt8\" (UID: \"41464378-d96a-4890-bcdb-fb23717b7593\") " pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.182062 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.182036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdsg\" (UniqueName: \"kubernetes.io/projected/2b4cea60-f228-49bb-b0d7-5122d386a695-kube-api-access-lxdsg\") pod \"ovnkube-node-f99c7\" (UID: \"2b4cea60-f228-49bb-b0d7-5122d386a695\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.185550 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.185520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stf9c\" (UniqueName: \"kubernetes.io/projected/ed08e791-3ce9-4d63-8283-489d4fd655e6-kube-api-access-stf9c\") pod \"iptables-alerter-9m2fz\" (UID: \"ed08e791-3ce9-4d63-8283-489d4fd655e6\") " pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.185822 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.185789 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtvg\" (UniqueName: \"kubernetes.io/projected/ff93e08a-42e5-457b-96c2-a54192e5518b-kube-api-access-sqtvg\") pod \"node-resolver-5mfzp\" (UID: \"ff93e08a-42e5-457b-96c2-a54192e5518b\") " pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.249033 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.248986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l85qb" Jul 30 20:21:04.264062 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.264027 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" Jul 30 20:21:04.272189 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.272160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:04.277287 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.277261 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" Jul 30 20:21:04.287300 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.287278 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5mfzp" Jul 30 20:21:04.297396 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.297371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ktn4b" Jul 30 20:21:04.305207 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.305179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" Jul 30 20:21:04.312317 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.312294 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:04.321329 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.321306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9m2fz" Jul 30 20:21:04.668512 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.668477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:04.668695 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.668539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:04.668695 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.668655 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:04.668819 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.668734 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret podName:19e1b2f5-58e3-423b-80d7-51670c463768 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:05.668714337 +0000 UTC m=+4.133852664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret") pod "global-pull-secret-syncer-6htjd" (UID: "19e1b2f5-58e3-423b-80d7-51670c463768") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:04.668819 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.668658 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:04.668924 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.668830 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs podName:dfc8c488-8fee-4f10-a041-8d67bc0569a8 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:05.668807039 +0000 UTC m=+4.133945378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs") pod "network-metrics-daemon-dgsbq" (UID: "dfc8c488-8fee-4f10-a041-8d67bc0569a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:04.769697 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.769662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:04.769861 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.769841 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:04.769900 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.769868 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:04.769900 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.769879 2575 projected.go:196] Error preparing data for projected volume kube-api-access-79c4s for pod openshift-network-diagnostics/network-check-target-k8266: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:04.769962 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:04.769950 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s podName:27a0cc53-05f6-454e-9feb-28dd6673da84 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:05.769935403 +0000 UTC m=+4.235073729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-79c4s" (UniqueName: "kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s") pod "network-check-target-k8266" (UID: "27a0cc53-05f6-454e-9feb-28dd6673da84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:04.989707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.989628 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 20:16:04 +0000 UTC" deadline="2028-04-23 17:38:09.858669284 +0000 UTC" Jul 30 20:21:04.989707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:04.989663 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15189h17m4.869010409s" Jul 30 20:21:05.006881 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:05.006847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:05.007079 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.006969 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:05.123757 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.123727 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda0552e_d5f2_4f26_9feb_5069fdbd32ff.slice/crio-a7ea29510e352b17bbe8fb0ca4ba183e68e1e28535d5adb3b56c341c75a8bc4e WatchSource:0}: Error finding container a7ea29510e352b17bbe8fb0ca4ba183e68e1e28535d5adb3b56c341c75a8bc4e: Status 404 returned error can't find the container with id a7ea29510e352b17bbe8fb0ca4ba183e68e1e28535d5adb3b56c341c75a8bc4e Jul 30 20:21:05.125421 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.125296 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41464378_d96a_4890_bcdb_fb23717b7593.slice/crio-9d3715b0c5084b3b1a8553f270b8acfe515a3d30b3b85e29e7f982e2e2f065c3 WatchSource:0}: Error finding container 9d3715b0c5084b3b1a8553f270b8acfe515a3d30b3b85e29e7f982e2e2f065c3: Status 404 returned error can't find the container with id 9d3715b0c5084b3b1a8553f270b8acfe515a3d30b3b85e29e7f982e2e2f065c3 Jul 30 20:21:05.126218 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.126094 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4343dc18_e764_4840_a67e_fa8389bf80eb.slice/crio-32da3b83760856210c75984ab646ba323b76025a5adad050ae5645c6b92627da WatchSource:0}: Error finding container 32da3b83760856210c75984ab646ba323b76025a5adad050ae5645c6b92627da: Status 404 returned error can't find the container with id 32da3b83760856210c75984ab646ba323b76025a5adad050ae5645c6b92627da Jul 30 20:21:05.129175 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.129158 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364bec14_8852_434b_a5d4_6d1bc081ad1d.slice/crio-97443193f4fd569c264703d7e3d341b9bf377741d7474d15bbf283f68a9cda5e WatchSource:0}: Error finding container 97443193f4fd569c264703d7e3d341b9bf377741d7474d15bbf283f68a9cda5e: Status 404 returned error can't find the container with id 97443193f4fd569c264703d7e3d341b9bf377741d7474d15bbf283f68a9cda5e Jul 30 20:21:05.130010 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.129988 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede1ef48_7bac_40e6_a395_7357ac99d373.slice/crio-1d50ccfe633a95390295cf30af16e8343d839aafe70017be7ede1a3e56be371a WatchSource:0}: Error finding container 1d50ccfe633a95390295cf30af16e8343d839aafe70017be7ede1a3e56be371a: Status 404 returned error can't find the container with id 1d50ccfe633a95390295cf30af16e8343d839aafe70017be7ede1a3e56be371a Jul 30 20:21:05.130815 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.130707 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded08e791_3ce9_4d63_8283_489d4fd655e6.slice/crio-528959125b880f28e7ced093b9e8da237347d88de51033491b86cb24ff496815 WatchSource:0}: Error finding container 528959125b880f28e7ced093b9e8da237347d88de51033491b86cb24ff496815: Status 404 returned error can't find the container with id 528959125b880f28e7ced093b9e8da237347d88de51033491b86cb24ff496815 Jul 30 20:21:05.132116 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.132094 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda966ad2_6153_4de3_8d61_4f80ce2fda88.slice/crio-3a2436e7d37c1e847a632e6680069ae12a7ba5cd4e63057c900a87a8de273d45 WatchSource:0}: Error finding container 3a2436e7d37c1e847a632e6680069ae12a7ba5cd4e63057c900a87a8de273d45: Status 404 returned error can't find the container with id 3a2436e7d37c1e847a632e6680069ae12a7ba5cd4e63057c900a87a8de273d45 Jul 30 20:21:05.132457 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.132387 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff93e08a_42e5_457b_96c2_a54192e5518b.slice/crio-b2b70d34357595b450ba09e88cabee826fdc91685bc291c15bff834677edc89e WatchSource:0}: Error finding container b2b70d34357595b450ba09e88cabee826fdc91685bc291c15bff834677edc89e: Status 404 returned error can't find the container with id b2b70d34357595b450ba09e88cabee826fdc91685bc291c15bff834677edc89e Jul 30 20:21:05.160838 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:05.160809 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4cea60_f228_49bb_b0d7_5122d386a695.slice/crio-b86267d142a82d66691d1138d2081d403a997fe6e378fbbc374c54e56d5f281b WatchSource:0}: Error finding container b86267d142a82d66691d1138d2081d403a997fe6e378fbbc374c54e56d5f281b: Status 404 returned error can't find the container with id b86267d142a82d66691d1138d2081d403a997fe6e378fbbc374c54e56d5f281b Jul 30 20:21:05.676874 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:05.676555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:05.676874 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:05.676844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:05.677121 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.676964 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:05.677121 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.677040 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs podName:dfc8c488-8fee-4f10-a041-8d67bc0569a8 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:07.677017056 +0000 UTC m=+6.142155385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs") pod "network-metrics-daemon-dgsbq" (UID: "dfc8c488-8fee-4f10-a041-8d67bc0569a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:05.677235 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.677166 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:05.677235 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.677205 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret podName:19e1b2f5-58e3-423b-80d7-51670c463768 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:07.677193399 +0000 UTC m=+6.142331729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret") pod "global-pull-secret-syncer-6htjd" (UID: "19e1b2f5-58e3-423b-80d7-51670c463768") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:05.777240 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:05.777196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:05.777433 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.777416 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:05.777504 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.777438 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:05.777504 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.777452 2575 projected.go:196] Error preparing data for projected volume kube-api-access-79c4s for pod openshift-network-diagnostics/network-check-target-k8266: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:05.777607 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:05.777511 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s podName:27a0cc53-05f6-454e-9feb-28dd6673da84 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:07.777491687 +0000 UTC m=+6.242630027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-79c4s" (UniqueName: "kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s") pod "network-check-target-k8266" (UID: "27a0cc53-05f6-454e-9feb-28dd6673da84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:05.989931 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:05.989796 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 20:16:04 +0000 UTC" deadline="2028-05-17 19:01:36.738792352 +0000 UTC" Jul 30 20:21:05.989931 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:05.989838 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15766h40m30.748958387s" Jul 30 20:21:06.009011 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.008330 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:06.009011 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:06.008466 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:06.009011 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.008852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:06.009011 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:06.008955 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:06.028368 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.028329 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" event={"ID":"c79bb9ccdd4a990748877883d79006b3","Type":"ContainerStarted","Data":"57c7d18cab06137e83d07bff2f03bdd93c3f7f3783e2ceac60e8f3b09f971b4c"} Jul 30 20:21:06.033762 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.033659 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-st6vb" event={"ID":"ede1ef48-7bac-40e6-a395-7357ac99d373","Type":"ContainerStarted","Data":"1d50ccfe633a95390295cf30af16e8343d839aafe70017be7ede1a3e56be371a"} Jul 30 20:21:06.056184 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.056147 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"b86267d142a82d66691d1138d2081d403a997fe6e378fbbc374c54e56d5f281b"} Jul 30 20:21:06.059924 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.059843 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9m2fz" event={"ID":"ed08e791-3ce9-4d63-8283-489d4fd655e6","Type":"ContainerStarted","Data":"528959125b880f28e7ced093b9e8da237347d88de51033491b86cb24ff496815"} Jul 30 20:21:06.064900 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.064870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l85qb" event={"ID":"4343dc18-e764-4840-a67e-fa8389bf80eb","Type":"ContainerStarted","Data":"32da3b83760856210c75984ab646ba323b76025a5adad050ae5645c6b92627da"} Jul 30 20:21:06.069036 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.068396 2575 generic.go:356] "Generic (PLEG): container finished" podID="e8c674c09deef1f48c2b8c515f7e6e86" containerID="fc2def90dafadbc342e6a1866e1b042c2a949c20365d37a580a0381b806721ad" exitCode=0 Jul 30 20:21:06.069036 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.068482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" event={"ID":"e8c674c09deef1f48c2b8c515f7e6e86","Type":"ContainerDied","Data":"fc2def90dafadbc342e6a1866e1b042c2a949c20365d37a580a0381b806721ad"} Jul 30 20:21:06.085938 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.085821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5mfzp" event={"ID":"ff93e08a-42e5-457b-96c2-a54192e5518b","Type":"ContainerStarted","Data":"b2b70d34357595b450ba09e88cabee826fdc91685bc291c15bff834677edc89e"} Jul 30 20:21:06.090390 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.090345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ktn4b" event={"ID":"da966ad2-6153-4de3-8d61-4f80ce2fda88","Type":"ContainerStarted","Data":"3a2436e7d37c1e847a632e6680069ae12a7ba5cd4e63057c900a87a8de273d45"} Jul 30 20:21:06.095864 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.095777 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-21.ec2.internal" podStartSLOduration=3.095759452 podStartE2EDuration="3.095759452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:06.045520793 +0000 UTC m=+4.510659164" watchObservedRunningTime="2026-07-30 20:21:06.095759452 +0000 UTC m=+4.560897804" Jul 30 20:21:06.098956 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.098930 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" event={"ID":"364bec14-8852-434b-a5d4-6d1bc081ad1d","Type":"ContainerStarted","Data":"97443193f4fd569c264703d7e3d341b9bf377741d7474d15bbf283f68a9cda5e"} Jul 30 20:21:06.116404 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.116296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" event={"ID":"41464378-d96a-4890-bcdb-fb23717b7593","Type":"ContainerStarted","Data":"9d3715b0c5084b3b1a8553f270b8acfe515a3d30b3b85e29e7f982e2e2f065c3"} Jul 30 20:21:06.118909 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:06.118878 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" event={"ID":"bda0552e-d5f2-4f26-9feb-5069fdbd32ff","Type":"ContainerStarted","Data":"a7ea29510e352b17bbe8fb0ca4ba183e68e1e28535d5adb3b56c341c75a8bc4e"} Jul 30 20:21:07.007462 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:07.007422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:07.007909 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.007558 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:07.136168 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:07.136125 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" event={"ID":"e8c674c09deef1f48c2b8c515f7e6e86","Type":"ContainerStarted","Data":"13080047fce4c66b5f723e2ae09cd70db2440300b2656250c14baa657f3f4ff0"} Jul 30 20:21:07.162479 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:07.162419 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-21.ec2.internal" podStartSLOduration=4.162398682 podStartE2EDuration="4.162398682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:07.16183846 +0000 UTC m=+5.626976808" watchObservedRunningTime="2026-07-30 20:21:07.162398682 +0000 UTC m=+5.627537031" Jul 30 20:21:07.702074 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:07.702030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:07.702271 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:07.702123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:07.702271 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.702226 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:07.702271 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.702245 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:07.702430 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.702297 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret podName:19e1b2f5-58e3-423b-80d7-51670c463768 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:11.702280603 +0000 UTC m=+10.167418942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret") pod "global-pull-secret-syncer-6htjd" (UID: "19e1b2f5-58e3-423b-80d7-51670c463768") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:07.702430 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.702317 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs podName:dfc8c488-8fee-4f10-a041-8d67bc0569a8 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:11.702307151 +0000 UTC m=+10.167445477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs") pod "network-metrics-daemon-dgsbq" (UID: "dfc8c488-8fee-4f10-a041-8d67bc0569a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:07.802995 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:07.802952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:07.803186 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.803167 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:07.803241 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.803194 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:07.803241 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.803208 2575 projected.go:196] Error preparing data for projected volume kube-api-access-79c4s for pod openshift-network-diagnostics/network-check-target-k8266: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:07.803353 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:07.803272 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s podName:27a0cc53-05f6-454e-9feb-28dd6673da84 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:11.803253598 +0000 UTC m=+10.268391947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-79c4s" (UniqueName: "kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s") pod "network-check-target-k8266" (UID: "27a0cc53-05f6-454e-9feb-28dd6673da84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:08.007479 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:08.007404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:08.007896 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:08.007546 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:08.007896 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:08.007883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:08.008020 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:08.007971 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:09.007713 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:09.007678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:09.008324 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:09.007809 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:10.007759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:10.007681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:10.008234 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:10.007823 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:10.008313 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:10.008291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:10.008432 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:10.008410 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:11.007184 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:11.006956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:11.007184 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.007087 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:11.736705 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:11.736497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:11.736705 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:11.736573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:11.736705 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.736677 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:11.737290 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.736762 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret podName:19e1b2f5-58e3-423b-80d7-51670c463768 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:19.736740586 +0000 UTC m=+18.201878913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret") pod "global-pull-secret-syncer-6htjd" (UID: "19e1b2f5-58e3-423b-80d7-51670c463768") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:11.737290 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.736683 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:11.737290 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.737137 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs podName:dfc8c488-8fee-4f10-a041-8d67bc0569a8 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:19.737121679 +0000 UTC m=+18.202260011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs") pod "network-metrics-daemon-dgsbq" (UID: "dfc8c488-8fee-4f10-a041-8d67bc0569a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:11.837573 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:11.837516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:11.837763 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.837707 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:11.837763 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.837728 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:11.837763 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.837740 2575 projected.go:196] Error preparing data for projected volume kube-api-access-79c4s for pod openshift-network-diagnostics/network-check-target-k8266: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:11.837915 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:11.837796 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s podName:27a0cc53-05f6-454e-9feb-28dd6673da84 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:19.837779177 +0000 UTC m=+18.302917517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-79c4s" (UniqueName: "kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s") pod "network-check-target-k8266" (UID: "27a0cc53-05f6-454e-9feb-28dd6673da84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:12.008329 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:12.008246 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:12.008487 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:12.008360 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:12.008487 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:12.008402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:12.008591 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:12.008522 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:13.007587 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:13.007318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:13.007587 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:13.007456 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:14.007278 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:14.007240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:14.007455 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:14.007385 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:14.007455 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:14.007443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:14.007594 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:14.007572 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:15.007609 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:15.007578 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:15.008122 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:15.008093 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:16.007198 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:16.007161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:16.007385 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:16.007170 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:16.007385 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:16.007321 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:16.007476 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:16.007382 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:17.007538 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:17.007509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:17.007985 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:17.007628 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:18.007455 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:18.007418 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:18.007455 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:18.007440 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:18.008012 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:18.007556 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:18.008012 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:18.007731 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:19.007149 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:19.007113 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:19.007357 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.007228 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:19.791513 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:19.791467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:19.791928 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:19.791531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:19.791928 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.791628 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:19.791928 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.791686 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs podName:dfc8c488-8fee-4f10-a041-8d67bc0569a8 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:35.791671931 +0000 UTC m=+34.256810257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs") pod "network-metrics-daemon-dgsbq" (UID: "dfc8c488-8fee-4f10-a041-8d67bc0569a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:19.791928 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.791628 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:19.791928 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.791776 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret podName:19e1b2f5-58e3-423b-80d7-51670c463768 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:35.791762453 +0000 UTC m=+34.256900787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret") pod "global-pull-secret-syncer-6htjd" (UID: "19e1b2f5-58e3-423b-80d7-51670c463768") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:19.892489 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:19.892445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:19.892662 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.892610 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:19.892735 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.892638 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:19.892735 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.892680 2575 projected.go:196] Error preparing data for projected volume kube-api-access-79c4s for pod openshift-network-diagnostics/network-check-target-k8266: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:19.892833 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:19.892756 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s podName:27a0cc53-05f6-454e-9feb-28dd6673da84 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:35.89273562 +0000 UTC m=+34.357873950 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-79c4s" (UniqueName: "kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s") pod "network-check-target-k8266" (UID: "27a0cc53-05f6-454e-9feb-28dd6673da84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:20.007038 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:20.007008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:20.007038 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:20.007024 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:20.007258 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:20.007134 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:20.007323 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:20.007290 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:21.007319 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:21.007285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:21.007763 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:21.007415 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:22.007731 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:22.007696 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:22.008262 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:22.007805 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:22.008262 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:22.007860 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:22.008262 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:22.007994 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:23.007747 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:23.007715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:23.008203 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:23.007834 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:24.007599 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.007298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:24.007755 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.007293 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:24.007755 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:24.007678 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:24.008380 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:24.007759 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:24.168775 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.168736 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" event={"ID":"bda0552e-d5f2-4f26-9feb-5069fdbd32ff","Type":"ContainerStarted","Data":"b58b44b6d2ef7ddea6e9c1877f5bb28975cbb6018d1f2fd72099542113faeb24"} Jul 30 20:21:24.170410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.170134 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-st6vb" event={"ID":"ede1ef48-7bac-40e6-a395-7357ac99d373","Type":"ContainerStarted","Data":"80443024c9b0d5f33f7bad4986b3119b333935ade0706e80a5743eefe043d062"} Jul 30 20:21:24.173556 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.173517 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"ac847ac679dcbbbd238ae7160dea35c14dd32cee73b93151d0d5ef0e57a88fdd"} Jul 30 20:21:24.173627 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.173615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"6dc4786d3b8ecd456d17fe9002dbe05b5344be2db7093019311c8c5d666b2403"} Jul 30 20:21:24.173733 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.173676 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"59eb982cdb4981e9412beb18f5bfe2f4115e6d17ce3e5a2301511a797ee34311"} Jul 30 20:21:24.173733 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.173715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"d93f20e4892ccd2b8cc554d3bba29d8f678d44e1e619e54e65e6936677ed4197"} Jul 30 20:21:24.175236 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.175210 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l85qb" event={"ID":"4343dc18-e764-4840-a67e-fa8389bf80eb","Type":"ContainerStarted","Data":"a0e40a038725bea135ef2ed617399ae6eb52b65472e758cdd22831eb39959e6f"} Jul 30 20:21:24.177841 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.177808 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5mfzp" event={"ID":"ff93e08a-42e5-457b-96c2-a54192e5518b","Type":"ContainerStarted","Data":"c42d48b6320966805be652f824c9cc1cfc88a2651e726f716077a274d630d558"} Jul 30 20:21:24.179701 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.179673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ktn4b" event={"ID":"da966ad2-6153-4de3-8d61-4f80ce2fda88","Type":"ContainerStarted","Data":"73cafead5f3d013080d25b4b70180ef6f07c690f0498d7fa41aaa44829e0cd40"} Jul 30 20:21:24.181487 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.181459 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" event={"ID":"364bec14-8852-434b-a5d4-6d1bc081ad1d","Type":"ContainerStarted","Data":"5806739cffde7d3bf4cbadce53aaace23faa66b33644368cb702caed3a89b0ea"} Jul 30 20:21:24.183366 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.183337 2575 generic.go:356] "Generic (PLEG): container finished" podID="41464378-d96a-4890-bcdb-fb23717b7593" containerID="704f8c6494d80854ec0509822ee2d03373cfb8f3a7cff6c3c0aa8c1495f389e6" exitCode=0 Jul 30 20:21:24.183471 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.183378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" event={"ID":"41464378-d96a-4890-bcdb-fb23717b7593","Type":"ContainerDied","Data":"704f8c6494d80854ec0509822ee2d03373cfb8f3a7cff6c3c0aa8c1495f389e6"} Jul 30 20:21:24.212192 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.212139 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-st6vb" podStartSLOduration=11.72833604 podStartE2EDuration="22.212123677s" totalImagesPullingTime="10.483787637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:24.19138381 +0000 UTC m=+22.656522171" watchObservedRunningTime="2026-07-30 20:21:24.212123677 +0000 UTC m=+22.677262024" Jul 30 20:21:24.212387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.212236 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vxfr2" podStartSLOduration=3.791926687 podStartE2EDuration="22.21222927s" totalImagesPullingTime="18.420302583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:24.211620466 +0000 UTC m=+22.676758815" watchObservedRunningTime="2026-07-30 20:21:24.21222927 +0000 UTC m=+22.677367621" Jul 30 20:21:24.230746 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.230697 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l85qb" podStartSLOduration=3.753468357 podStartE2EDuration="22.230667085s" totalImagesPullingTime="18.477198728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:24.230337272 +0000 UTC m=+22.695475659" watchObservedRunningTime="2026-07-30 20:21:24.230667085 +0000 UTC m=+22.695805433" Jul 30 20:21:24.250175 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.250117 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5mfzp" podStartSLOduration=3.8618616279999998 podStartE2EDuration="22.250097609s" totalImagesPullingTime="18.388235981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:24.249718952 +0000 UTC m=+22.714857320" watchObservedRunningTime="2026-07-30 20:21:24.250097609 +0000 UTC m=+22.715235957" Jul 30 20:21:24.336246 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:24.336132 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ktn4b" podStartSLOduration=11.852174251 podStartE2EDuration="22.336109808s" totalImagesPullingTime="10.483935557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:24.282981121 +0000 UTC m=+22.748119490" watchObservedRunningTime="2026-07-30 20:21:24.336109808 +0000 UTC m=+22.801248155" Jul 30 20:21:25.007549 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.007520 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:25.007682 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:25.007626 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:25.013762 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.013734 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 20:21:25.113264 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.113162 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T20:21:25.013753313Z","UUID":"4a9a343b-a22d-42f4-9b95-4ded55b2f4d6","Handler":null,"Name":"","Endpoint":""} Jul 30 20:21:25.115107 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.115083 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 20:21:25.115237 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.115127 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 20:21:25.188041 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.187991 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" event={"ID":"bda0552e-d5f2-4f26-9feb-5069fdbd32ff","Type":"ContainerStarted","Data":"1cb798398f540824a2164d7fe95ed58e181254d4619881ac8a8538f3b503a0a3"} Jul 30 20:21:25.191821 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.191792 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"2cbc8ac6fd52dd6afb3815b31508466c3624c9d12f0f8790a9f6e8971dab5f74"} Jul 30 20:21:25.191984 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.191875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"b903ba56f065ce19428312f4500a97b27e44fc0e29b741adc47c9cedf0441f26"} Jul 30 20:21:25.193540 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:25.193498 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9m2fz" event={"ID":"ed08e791-3ce9-4d63-8283-489d4fd655e6","Type":"ContainerStarted","Data":"b142a7dd7f14d4baf01b1dfaf3cf9feb9bc8788835b699145504b13bc51d7428"} Jul 30 20:21:26.007732 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:26.007694 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:26.008148 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:26.007847 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:26.008148 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:26.007880 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:26.008148 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:26.008010 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:26.225001 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:26.224959 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:26.225723 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:26.225700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:26.259408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:26.259230 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9m2fz" podStartSLOduration=5.869285023 podStartE2EDuration="24.259214685s" totalImagesPullingTime="18.389929662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:25.220952101 +0000 UTC m=+23.686090451" watchObservedRunningTime="2026-07-30 20:21:26.259214685 +0000 UTC m=+24.724353033" Jul 30 20:21:27.007518 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:27.007492 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:27.007801 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:27.007596 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:27.200568 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:27.200533 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" event={"ID":"bda0552e-d5f2-4f26-9feb-5069fdbd32ff","Type":"ContainerStarted","Data":"ae4683c667815b155f0db252a4802091586f271a345317aeb27f61a33fc969a5"} Jul 30 20:21:27.203384 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:27.203358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"093716da2cb47a4bdd71cb405de92ec6fe60c7840257460f58b115d027198145"} Jul 30 20:21:27.203577 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:27.203563 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:27.203989 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:27.203968 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-st6vb" Jul 30 20:21:27.235002 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:27.234966 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vpd5" podStartSLOduration=4.491416884 podStartE2EDuration="25.234952677s" totalImagesPullingTime="20.743535793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:27.21923393 +0000 UTC m=+25.684372288" watchObservedRunningTime="2026-07-30 20:21:27.234952677 +0000 UTC m=+25.700091025" Jul 30 20:21:28.010213 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:28.010180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:28.010391 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:28.010223 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:28.010391 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:28.010311 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:28.010506 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:28.010395 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:29.007228 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:29.007200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:29.007894 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:29.007299 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:29.208937 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:29.208902 2575 generic.go:356] "Generic (PLEG): container finished" podID="41464378-d96a-4890-bcdb-fb23717b7593" containerID="6dcf4834349bf5d62178f33864fce30af8f5bb8e535abdd713eec4a5678c6f77" exitCode=0 Jul 30 20:21:29.209119 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:29.208989 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" event={"ID":"41464378-d96a-4890-bcdb-fb23717b7593","Type":"ContainerDied","Data":"6dcf4834349bf5d62178f33864fce30af8f5bb8e535abdd713eec4a5678c6f77"} Jul 30 20:21:29.212683 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:29.212660 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" event={"ID":"2b4cea60-f228-49bb-b0d7-5122d386a695","Type":"ContainerStarted","Data":"a831dcf1660cfb5f2fe305d35c9c0e5c8f30c17fd063fe78cc8513a89d411d2b"} Jul 30 20:21:29.263812 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:29.263775 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" podStartSLOduration=8.7978083 podStartE2EDuration="27.263761266s" totalImagesPullingTime="18.465952966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:29.262606112 +0000 UTC m=+27.727744459" watchObservedRunningTime="2026-07-30 20:21:29.263761266 +0000 UTC m=+27.728899613" Jul 30 20:21:30.007960 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.007743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:30.008503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.007814 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:30.008503 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:30.008048 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:30.008503 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:30.008344 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:30.215907 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.215855 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:30.216122 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.215945 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:30.216122 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.215992 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:30.232537 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.232512 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:30.232660 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.232617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:21:30.787877 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.787847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k8266"] Jul 30 20:21:30.788088 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.787996 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:30.788163 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:30.788123 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:30.791403 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.791383 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6htjd"] Jul 30 20:21:30.791528 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.791514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:30.791639 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:30.791613 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:30.792113 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.792092 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dgsbq"] Jul 30 20:21:30.792209 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:30.792186 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:30.792297 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:30.792266 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:31.218770 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:31.218740 2575 generic.go:356] "Generic (PLEG): container finished" podID="41464378-d96a-4890-bcdb-fb23717b7593" containerID="34943ad82306d21a42f0728e1ae96436a273ac6afb71dbc18a8f6f2664e46d01" exitCode=0 Jul 30 20:21:31.219234 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:31.218826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" event={"ID":"41464378-d96a-4890-bcdb-fb23717b7593","Type":"ContainerDied","Data":"34943ad82306d21a42f0728e1ae96436a273ac6afb71dbc18a8f6f2664e46d01"} Jul 30 20:21:32.008854 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:32.008824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:32.009005 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:32.008954 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:33.007605 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:33.007578 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:33.007927 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:33.007679 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:33.007927 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:33.007729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:33.007927 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:33.007813 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:33.225110 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:33.225041 2575 generic.go:356] "Generic (PLEG): container finished" podID="41464378-d96a-4890-bcdb-fb23717b7593" containerID="3db9b3a9276ee805d3bbebbf2191792db40375b9b9833cebefcfba19bda386d1" exitCode=0 Jul 30 20:21:33.225110 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:33.225092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" event={"ID":"41464378-d96a-4890-bcdb-fb23717b7593","Type":"ContainerDied","Data":"3db9b3a9276ee805d3bbebbf2191792db40375b9b9833cebefcfba19bda386d1"} Jul 30 20:21:34.007142 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:34.007108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:34.007336 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:34.007240 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:35.007484 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:35.007453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:35.007889 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.007582 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6htjd" podUID="19e1b2f5-58e3-423b-80d7-51670c463768" Jul 30 20:21:35.007889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:35.007634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:35.007889 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.007744 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8266" podUID="27a0cc53-05f6-454e-9feb-28dd6673da84" Jul 30 20:21:35.806177 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:35.806148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:35.806352 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:35.806209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:35.806352 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.806307 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:35.806352 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.806315 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:35.806492 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.806369 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret podName:19e1b2f5-58e3-423b-80d7-51670c463768 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:07.806350611 +0000 UTC m=+66.271488936 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret") pod "global-pull-secret-syncer-6htjd" (UID: "19e1b2f5-58e3-423b-80d7-51670c463768") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:35.806492 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.806387 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs podName:dfc8c488-8fee-4f10-a041-8d67bc0569a8 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:07.806378015 +0000 UTC m=+66.271516341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs") pod "network-metrics-daemon-dgsbq" (UID: "dfc8c488-8fee-4f10-a041-8d67bc0569a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:35.907434 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:35.907402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:35.907589 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.907542 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:35.907589 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.907562 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:35.907589 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.907574 2575 projected.go:196] Error preparing data for projected volume kube-api-access-79c4s for pod openshift-network-diagnostics/network-check-target-k8266: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:35.907735 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:35.907628 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s podName:27a0cc53-05f6-454e-9feb-28dd6673da84 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:07.907611993 +0000 UTC m=+66.372750343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-79c4s" (UniqueName: "kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s") pod "network-check-target-k8266" (UID: "27a0cc53-05f6-454e-9feb-28dd6673da84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:36.007909 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.007880 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:36.008398 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.008010 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dgsbq" podUID="dfc8c488-8fee-4f10-a041-8d67bc0569a8" Jul 30 20:21:36.379752 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.379722 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-21.ec2.internal" event="NodeReady" Jul 30 20:21:36.379929 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.379836 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 20:21:36.418817 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.418741 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk"] Jul 30 20:21:36.424824 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.424800 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-wbkxk"] Jul 30 20:21:36.425184 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.425161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:36.428196 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.428164 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv"] Jul 30 20:21:36.428312 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.428264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.431536 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.431515 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c89bb85d8-xz8kr"] Jul 30 20:21:36.431667 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.431650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.434790 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.434765 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j"] Jul 30 20:21:36.435607 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.435408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.437987 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.437968 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc"] Jul 30 20:21:36.438090 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.438076 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.440920 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.440898 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 30 20:21:36.441036 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.440906 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.441544 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.441526 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bb5fff7f9-2fsh7"] Jul 30 20:21:36.441674 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.441618 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.444582 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.444563 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj"] Jul 30 20:21:36.444687 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.444650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.446457 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.446437 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 30 20:21:36.446663 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.446646 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.446782 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.446765 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 30 20:21:36.446887 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.446775 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.446887 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.446688 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.447737 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.447684 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.447948 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.447933 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 30 20:21:36.448247 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.447992 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.448337 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.448005 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ct4r5\"" Jul 30 20:21:36.448539 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.448520 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-4lfdb"] Jul 30 20:21:36.449104 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.449079 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj" Jul 30 20:21:36.449811 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.449523 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 30 20:21:36.449811 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.449539 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 30 20:21:36.449811 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.449574 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 30 20:21:36.449811 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.449770 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 30 20:21:36.450082 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.449954 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.450082 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.450005 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 30 20:21:36.450310 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.450188 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.450310 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.450248 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.450470 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.450456 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.452844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.451366 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 30 20:21:36.452844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.451511 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 30 20:21:36.452844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.451584 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mjc5x\"" Jul 30 20:21:36.452844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.451696 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 30 20:21:36.452844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.452668 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 30 20:21:36.453145 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.453123 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk"] Jul 30 20:21:36.453200 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.453148 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-wbkxk"] Jul 30 20:21:36.453248 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.453210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.453458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.453385 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 20:21:36.455109 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.454295 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gllm5\"" Jul 30 20:21:36.455109 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.454447 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q4l4z\"" Jul 30 20:21:36.455109 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.454575 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.455109 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.454702 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6g6j5\"" Jul 30 20:21:36.455109 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.454816 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xkxnt\"" Jul 30 20:21:36.455109 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.454951 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-x28w6\"" Jul 30 20:21:36.455435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.455278 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2nln4\"" Jul 30 20:21:36.455540 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.455501 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv"] Jul 30 20:21:36.455540 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.455528 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 20:21:36.455872 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.455700 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.455958 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.455896 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc"] Jul 30 20:21:36.457393 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.457318 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.458617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.457849 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.458617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.457963 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 20:21:36.458617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.458091 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.458617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.458262 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.458617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.458293 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-s8jkp\"" Jul 30 20:21:36.458617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.457855 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c89bb85d8-xz8kr"] Jul 30 20:21:36.458617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.458484 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 30 20:21:36.458617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.458494 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 30 20:21:36.467470 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.467441 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 30 20:21:36.468439 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.468416 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 20:21:36.469889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.469838 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 30 20:21:36.471594 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.471399 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj"] Jul 30 20:21:36.472328 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.472304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-4lfdb"] Jul 30 20:21:36.474079 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.474033 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx"] Jul 30 20:21:36.479009 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.478990 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wxt4p"] Jul 30 20:21:36.479119 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.479109 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx" Jul 30 20:21:36.482504 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.482354 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nvthr"] Jul 30 20:21:36.482572 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.482505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxt4p" Jul 30 20:21:36.484220 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.484201 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.485405 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.485385 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 20:21:36.485788 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.485761 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7hfpr\"" Jul 30 20:21:36.485960 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.485946 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.486037 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.486003 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb5fff7f9-2fsh7"] Jul 30 20:21:36.486118 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.486037 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j"] Jul 30 20:21:36.486118 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.486104 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 20:21:36.486213 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.486148 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.486213 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.485307 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mc5cv\"" Jul 30 20:21:36.486644 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.486628 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 20:21:36.491235 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.491213 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 20:21:36.492342 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.492323 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxt4p"] Jul 30 20:21:36.493756 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.493735 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 20:21:36.493756 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.493747 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nmld7\"" Jul 30 20:21:36.494748 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.494728 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx"] Jul 30 20:21:36.505232 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.505190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nvthr"] Jul 30 20:21:36.511413 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.511392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:36.511613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.511590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcxf\" (UniqueName: \"kubernetes.io/projected/6878b2de-cd48-439d-859d-38e28a60e621-kube-api-access-7lcxf\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:36.612936 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.612902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84cde953-2d57-4ce5-8848-8079a5023664-snapshots\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.613121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.612945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c96510c-70a4-411f-952b-1a4176ab3094-config\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.613121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.612974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.613121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.612998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-bound-sa-token\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.613121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.613121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-trusted-ca\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.613121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svqvb\" (UniqueName: \"kubernetes.io/projected/a8624b77-b71d-41ac-a096-ec848d11f772-kube-api-access-svqvb\") pod \"network-check-source-65bdbbff45-kshwx\" (UID: \"a8624b77-b71d-41ac-a096-ec848d11f772\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thp4v\" (UniqueName: \"kubernetes.io/projected/84cde953-2d57-4ce5-8848-8079a5023664-kube-api-access-thp4v\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84cde953-2d57-4ce5-8848-8079a5023664-tmp\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xxc\" (UniqueName: \"kubernetes.io/projected/277d11be-7c47-46e8-8d2a-5e4016c305a7-kube-api-access-x4xxc\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kls2b\" (UniqueName: \"kubernetes.io/projected/fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685-kube-api-access-kls2b\") pod \"ingress-canary-wxt4p\" (UID: \"fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685\") " pod="openshift-ingress-canary/ingress-canary-wxt4p" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0f10072-fc2c-4d23-91b3-1a982a9decbd-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2bn\" (UniqueName: \"kubernetes.io/projected/a0f10072-fc2c-4d23-91b3-1a982a9decbd-kube-api-access-kf2bn\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613306 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-serving-cert\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-certificates\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84cde953-2d57-4ce5-8848-8079a5023664-service-ca-bundle\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84cde953-2d57-4ce5-8848-8079a5023664-trusted-ca-bundle\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.613410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnp8\" (UniqueName: \"kubernetes.io/projected/9c96510c-70a4-411f-952b-1a4176ab3094-kube-api-access-4hnp8\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-stats-auth\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcxf\" (UniqueName: \"kubernetes.io/projected/6878b2de-cd48-439d-859d-38e28a60e621-kube-api-access-7lcxf\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685-cert\") pod \"ingress-canary-wxt4p\" (UID: \"fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685\") " pod="openshift-ingress-canary/ingress-canary-wxt4p" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277d11be-7c47-46e8-8d2a-5e4016c305a7-tmp-dir\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ba3571c-c95e-4bc3-a238-eee309a1b456-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-image-registry-private-configuration\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/277d11be-7c47-46e8-8d2a-5e4016c305a7-config-volume\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0f10072-fc2c-4d23-91b3-1a982a9decbd-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84cde953-2d57-4ce5-8848-8079a5023664-serving-cert\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-config\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-trusted-ca\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6c4\" (UniqueName: \"kubernetes.io/projected/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-kube-api-access-6c6c4\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.613980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b78f9503-3ff3-4800-92ef-da6471ad6071-ca-trust-extracted\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnf22\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-kube-api-access-jnf22\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fdpr\" (UniqueName: \"kubernetes.io/projected/7ba3571c-c95e-4bc3-a238-eee309a1b456-kube-api-access-5fdpr\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.613960 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.613968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p9dn\" (UniqueName: \"kubernetes.io/projected/0f6c52e7-2883-4cc4-9eca-d526502011a5-kube-api-access-7p9dn\") pod \"volume-data-source-validator-5b8776669-242bj\" (UID: \"0f6c52e7-2883-4cc4-9eca-d526502011a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.614037 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls podName:6878b2de-cd48-439d-859d-38e28a60e621 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:37.114018602 +0000 UTC m=+35.579156929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-lfjkk" (UID: "6878b2de-cd48-439d-859d-38e28a60e621") : secret "samples-operator-tls" not found Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.614092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-default-certificate\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.614116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4982p\" (UniqueName: \"kubernetes.io/projected/7c5e8c92-080a-4b16-844a-e3310c5dc43f-kube-api-access-4982p\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.614133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-installation-pull-secrets\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.614719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.614164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c96510c-70a4-411f-952b-1a4176ab3094-serving-cert\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.631161 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.631131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcxf\" (UniqueName: \"kubernetes.io/projected/6878b2de-cd48-439d-859d-38e28a60e621-kube-api-access-7lcxf\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:36.715432 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c96510c-70a4-411f-952b-1a4176ab3094-serving-cert\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.715598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84cde953-2d57-4ce5-8848-8079a5023664-snapshots\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.715598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c96510c-70a4-411f-952b-1a4176ab3094-config\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.715598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.715598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-bound-sa-token\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.715598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.715598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-trusted-ca\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.715598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svqvb\" (UniqueName: \"kubernetes.io/projected/a8624b77-b71d-41ac-a096-ec848d11f772-kube-api-access-svqvb\") pod \"network-check-source-65bdbbff45-kshwx\" (UID: \"a8624b77-b71d-41ac-a096-ec848d11f772\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thp4v\" (UniqueName: \"kubernetes.io/projected/84cde953-2d57-4ce5-8848-8079a5023664-kube-api-access-thp4v\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84cde953-2d57-4ce5-8848-8079a5023664-tmp\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xxc\" (UniqueName: \"kubernetes.io/projected/277d11be-7c47-46e8-8d2a-5e4016c305a7-kube-api-access-x4xxc\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kls2b\" (UniqueName: \"kubernetes.io/projected/fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685-kube-api-access-kls2b\") pod \"ingress-canary-wxt4p\" (UID: \"fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685\") " pod="openshift-ingress-canary/ingress-canary-wxt4p" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0f10072-fc2c-4d23-91b3-1a982a9decbd-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2bn\" (UniqueName: \"kubernetes.io/projected/a0f10072-fc2c-4d23-91b3-1a982a9decbd-kube-api-access-kf2bn\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-serving-cert\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-certificates\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84cde953-2d57-4ce5-8848-8079a5023664-service-ca-bundle\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84cde953-2d57-4ce5-8848-8079a5023664-trusted-ca-bundle\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnp8\" (UniqueName: \"kubernetes.io/projected/9c96510c-70a4-411f-952b-1a4176ab3094-kube-api-access-4hnp8\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.715939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-stats-auth\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c96510c-70a4-411f-952b-1a4176ab3094-config\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.715991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685-cert\") pod \"ingress-canary-wxt4p\" (UID: \"fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685\") " pod="openshift-ingress-canary/ingress-canary-wxt4p" Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716074 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84cde953-2d57-4ce5-8848-8079a5023664-snapshots\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277d11be-7c47-46e8-8d2a-5e4016c305a7-tmp-dir\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ba3571c-c95e-4bc3-a238-eee309a1b456-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-image-registry-private-configuration\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.716172 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.716192 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5fff7f9-2fsh7: secret "image-registry-tls" not found Jul 30 20:21:36.716515 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.716265 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls podName:b78f9503-3ff3-4800-92ef-da6471ad6071 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:37.216245548 +0000 UTC m=+35.681383891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls") pod "image-registry-5bb5fff7f9-2fsh7" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071") : secret "image-registry-tls" not found Jul 30 20:21:36.716988 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84cde953-2d57-4ce5-8848-8079a5023664-tmp\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.716988 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716616 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-certificates\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.716988 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/277d11be-7c47-46e8-8d2a-5e4016c305a7-config-volume\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.716988 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84cde953-2d57-4ce5-8848-8079a5023664-service-ca-bundle\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.717202 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.716178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/277d11be-7c47-46e8-8d2a-5e4016c305a7-config-volume\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.717202 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.717202 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0f10072-fc2c-4d23-91b3-1a982a9decbd-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.717202 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84cde953-2d57-4ce5-8848-8079a5023664-serving-cert\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.717446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-config\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.717446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-trusted-ca\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.717446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6c4\" (UniqueName: \"kubernetes.io/projected/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-kube-api-access-6c6c4\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.717446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b78f9503-3ff3-4800-92ef-da6471ad6071-ca-trust-extracted\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.717446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnf22\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-kube-api-access-jnf22\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.717446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.717446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fdpr\" (UniqueName: \"kubernetes.io/projected/7ba3571c-c95e-4bc3-a238-eee309a1b456-kube-api-access-5fdpr\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.717446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-trusted-ca\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.717831 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9dn\" (UniqueName: \"kubernetes.io/projected/0f6c52e7-2883-4cc4-9eca-d526502011a5-kube-api-access-7p9dn\") pod \"volume-data-source-validator-5b8776669-242bj\" (UID: \"0f6c52e7-2883-4cc4-9eca-d526502011a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj" Jul 30 20:21:36.717831 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-default-certificate\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.717831 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4982p\" (UniqueName: \"kubernetes.io/projected/7c5e8c92-080a-4b16-844a-e3310c5dc43f-kube-api-access-4982p\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.717831 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-installation-pull-secrets\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.717831 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ba3571c-c95e-4bc3-a238-eee309a1b456-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.717831 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.717746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84cde953-2d57-4ce5-8848-8079a5023664-trusted-ca-bundle\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.718448 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.718335 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:37.218315683 +0000 UTC m=+35.683454026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : configmap references non-existent config key: service-ca.crt Jul 30 20:21:36.718448 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.718405 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:36.718448 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.718439 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls podName:277d11be-7c47-46e8-8d2a-5e4016c305a7 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:37.218426267 +0000 UTC m=+35.683564594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls") pod "dns-default-nvthr" (UID: "277d11be-7c47-46e8-8d2a-5e4016c305a7") : secret "dns-default-metrics-tls" not found Jul 30 20:21:36.718711 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.718676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277d11be-7c47-46e8-8d2a-5e4016c305a7-tmp-dir\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.718711 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.718692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0f10072-fc2c-4d23-91b3-1a982a9decbd-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.718711 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.718707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-trusted-ca\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.718911 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.718852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0f10072-fc2c-4d23-91b3-1a982a9decbd-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.718973 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.718918 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 20:21:36.718973 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.718948 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:37.218937971 +0000 UTC m=+35.684076297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : secret "router-metrics-certs-default" not found Jul 30 20:21:36.719571 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.719550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-config\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.719668 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.719568 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c96510c-70a4-411f-952b-1a4176ab3094-serving-cert\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.719668 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.719631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b78f9503-3ff3-4800-92ef-da6471ad6071-ca-trust-extracted\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.719668 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.719656 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:36.719840 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:36.719716 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls podName:7ba3571c-c95e-4bc3-a238-eee309a1b456 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:37.219700141 +0000 UTC m=+35.684838470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-xqtfv" (UID: "7ba3571c-c95e-4bc3-a238-eee309a1b456") : secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:36.720319 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.720293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685-cert\") pod \"ingress-canary-wxt4p\" (UID: \"fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685\") " pod="openshift-ingress-canary/ingress-canary-wxt4p" Jul 30 20:21:36.720415 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.720393 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-installation-pull-secrets\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.721193 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.721128 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84cde953-2d57-4ce5-8848-8079a5023664-serving-cert\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.721843 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.721819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-serving-cert\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.722540 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.722496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-default-certificate\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.722623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.722586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-image-registry-private-configuration\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.723725 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.723703 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-stats-auth\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.731807 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.731784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-bound-sa-token\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.731941 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.731921 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thp4v\" (UniqueName: \"kubernetes.io/projected/84cde953-2d57-4ce5-8848-8079a5023664-kube-api-access-thp4v\") pod \"insights-operator-77696685b-wbkxk\" (UID: \"84cde953-2d57-4ce5-8848-8079a5023664\") " pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.732068 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.732037 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svqvb\" (UniqueName: \"kubernetes.io/projected/a8624b77-b71d-41ac-a096-ec848d11f772-kube-api-access-svqvb\") pod \"network-check-source-65bdbbff45-kshwx\" (UID: \"a8624b77-b71d-41ac-a096-ec848d11f772\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx" Jul 30 20:21:36.734844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.734818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2bn\" (UniqueName: \"kubernetes.io/projected/a0f10072-fc2c-4d23-91b3-1a982a9decbd-kube-api-access-kf2bn\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xx7xc\" (UID: \"a0f10072-fc2c-4d23-91b3-1a982a9decbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.735375 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.735317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnf22\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-kube-api-access-jnf22\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:36.735375 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.735317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kls2b\" (UniqueName: \"kubernetes.io/projected/fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685-kube-api-access-kls2b\") pod \"ingress-canary-wxt4p\" (UID: \"fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685\") " pod="openshift-ingress-canary/ingress-canary-wxt4p" Jul 30 20:21:36.736358 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.736317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xxc\" (UniqueName: \"kubernetes.io/projected/277d11be-7c47-46e8-8d2a-5e4016c305a7-kube-api-access-x4xxc\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:36.736664 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.736642 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4982p\" (UniqueName: \"kubernetes.io/projected/7c5e8c92-080a-4b16-844a-e3310c5dc43f-kube-api-access-4982p\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:36.737250 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.737226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6c4\" (UniqueName: \"kubernetes.io/projected/6cd73e85-ac41-4b79-aa4d-3fcb090e046e-kube-api-access-6c6c4\") pod \"console-operator-5ccd6b445f-4lfdb\" (UID: \"6cd73e85-ac41-4b79-aa4d-3fcb090e046e\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.737348 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.737277 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fdpr\" (UniqueName: \"kubernetes.io/projected/7ba3571c-c95e-4bc3-a238-eee309a1b456-kube-api-access-5fdpr\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:36.739918 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.739896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9dn\" (UniqueName: \"kubernetes.io/projected/0f6c52e7-2883-4cc4-9eca-d526502011a5-kube-api-access-7p9dn\") pod \"volume-data-source-validator-5b8776669-242bj\" (UID: \"0f6c52e7-2883-4cc4-9eca-d526502011a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj" Jul 30 20:21:36.741404 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.741375 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnp8\" (UniqueName: \"kubernetes.io/projected/9c96510c-70a4-411f-952b-1a4176ab3094-kube-api-access-4hnp8\") pod \"service-ca-operator-784f97bcf6-9c65j\" (UID: \"9c96510c-70a4-411f-952b-1a4176ab3094\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.745556 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.745540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-wbkxk" Jul 30 20:21:36.779685 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.779664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" Jul 30 20:21:36.786156 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.786135 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" Jul 30 20:21:36.801131 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.801109 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj" Jul 30 20:21:36.810938 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.810921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:36.816656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.816637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx" Jul 30 20:21:36.823486 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:36.823460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxt4p" Jul 30 20:21:37.007708 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.007644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:21:37.007843 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.007651 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:21:37.010921 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.010893 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 20:21:37.011431 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.011417 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jh8wl\"" Jul 30 20:21:37.120533 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.120497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:37.120696 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.120670 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 20:21:37.120766 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.120743 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls podName:6878b2de-cd48-439d-859d-38e28a60e621 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:38.120721055 +0000 UTC m=+36.585859399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-lfjkk" (UID: "6878b2de-cd48-439d-859d-38e28a60e621") : secret "samples-operator-tls" not found Jul 30 20:21:37.221389 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.221349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:37.221540 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.221402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:37.221540 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221526 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:38.22150614 +0000 UTC m=+36.686644479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : configmap references non-existent config key: service-ca.crt Jul 30 20:21:37.221540 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221532 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 20:21:37.221699 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.221595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:37.221699 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221665 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:37.221699 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221685 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:38.221661203 +0000 UTC m=+36.686799736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : secret "router-metrics-certs-default" not found Jul 30 20:21:37.221841 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221714 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls podName:7ba3571c-c95e-4bc3-a238-eee309a1b456 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:38.221698417 +0000 UTC m=+36.686836743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-xqtfv" (UID: "7ba3571c-c95e-4bc3-a238-eee309a1b456") : secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:37.221841 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.221738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:37.221841 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.221775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:37.221997 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221861 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 20:21:37.221997 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221874 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:37.221997 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221877 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5fff7f9-2fsh7: secret "image-registry-tls" not found Jul 30 20:21:37.221997 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221909 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls podName:277d11be-7c47-46e8-8d2a-5e4016c305a7 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:38.221899603 +0000 UTC m=+36.687037929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls") pod "dns-default-nvthr" (UID: "277d11be-7c47-46e8-8d2a-5e4016c305a7") : secret "dns-default-metrics-tls" not found Jul 30 20:21:37.221997 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:37.221931 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls podName:b78f9503-3ff3-4800-92ef-da6471ad6071 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:38.221917603 +0000 UTC m=+36.687055940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls") pod "image-registry-5bb5fff7f9-2fsh7" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071") : secret "image-registry-tls" not found Jul 30 20:21:37.586009 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.585762 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx"] Jul 30 20:21:37.587305 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:37.587279 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8624b77_b71d_41ac_a096_ec848d11f772.slice/crio-b1173bdbfd6ca9131412f351498f2899dc94deeba6ff0f99cdadbe89f98bc5ca WatchSource:0}: Error finding container b1173bdbfd6ca9131412f351498f2899dc94deeba6ff0f99cdadbe89f98bc5ca: Status 404 returned error can't find the container with id b1173bdbfd6ca9131412f351498f2899dc94deeba6ff0f99cdadbe89f98bc5ca Jul 30 20:21:37.590198 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.590133 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj"] Jul 30 20:21:37.591842 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.591823 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxt4p"] Jul 30 20:21:37.599240 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.599210 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc"] Jul 30 20:21:37.601500 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.601480 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j"] Jul 30 20:21:37.609087 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.609048 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-wbkxk"] Jul 30 20:21:37.610188 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:37.610165 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cde953_2d57_4ce5_8848_8079a5023664.slice/crio-29ac7807ae6cee2cb7f53f21bc7441c40ee1d4dab9a72a779e71bc2784fc8dbd WatchSource:0}: Error finding container 29ac7807ae6cee2cb7f53f21bc7441c40ee1d4dab9a72a779e71bc2784fc8dbd: Status 404 returned error can't find the container with id 29ac7807ae6cee2cb7f53f21bc7441c40ee1d4dab9a72a779e71bc2784fc8dbd Jul 30 20:21:37.610292 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:37.610210 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-4lfdb"] Jul 30 20:21:37.611954 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:37.611924 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd73e85_ac41_4b79_aa4d_3fcb090e046e.slice/crio-1683c72f5a7c2d9cdf40e1c7dc2235d40b021d97c2921938c60f8e8c65a1b60e WatchSource:0}: Error finding container 1683c72f5a7c2d9cdf40e1c7dc2235d40b021d97c2921938c60f8e8c65a1b60e: Status 404 returned error can't find the container with id 1683c72f5a7c2d9cdf40e1c7dc2235d40b021d97c2921938c60f8e8c65a1b60e Jul 30 20:21:37.612740 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:37.612722 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8706b5_7f1b_4c67_bf3a_bd0a6c3d9685.slice/crio-87e501e14d713b99d8fd5e10956ab8a362db6dd35e36d9f73c63159da2e98f3f WatchSource:0}: Error finding container 87e501e14d713b99d8fd5e10956ab8a362db6dd35e36d9f73c63159da2e98f3f: Status 404 returned error can't find the container with id 87e501e14d713b99d8fd5e10956ab8a362db6dd35e36d9f73c63159da2e98f3f Jul 30 20:21:37.613489 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:37.613465 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c96510c_70a4_411f_952b_1a4176ab3094.slice/crio-e0155905ccfd6123470a62431a0e795edca5071e9832dff5c1204bf0e02e1d33 WatchSource:0}: Error finding container e0155905ccfd6123470a62431a0e795edca5071e9832dff5c1204bf0e02e1d33: Status 404 returned error can't find the container with id e0155905ccfd6123470a62431a0e795edca5071e9832dff5c1204bf0e02e1d33 Jul 30 20:21:37.614141 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:37.614120 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6c52e7_2883_4cc4_9eca_d526502011a5.slice/crio-8b3d885d163f0d8d75afbea3c296e76750e3da0675024e06d0f9994413d61a70 WatchSource:0}: Error finding container 8b3d885d163f0d8d75afbea3c296e76750e3da0675024e06d0f9994413d61a70: Status 404 returned error can't find the container with id 8b3d885d163f0d8d75afbea3c296e76750e3da0675024e06d0f9994413d61a70 Jul 30 20:21:37.615479 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:37.615408 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f10072_fc2c_4d23_91b3_1a982a9decbd.slice/crio-8b397effb4544d8d00b226d25844d6a5a62690511ea80c65c93ed1239e085061 WatchSource:0}: Error finding container 8b397effb4544d8d00b226d25844d6a5a62690511ea80c65c93ed1239e085061: Status 404 returned error can't find the container with id 8b397effb4544d8d00b226d25844d6a5a62690511ea80c65c93ed1239e085061 Jul 30 20:21:38.007672 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.007639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:21:38.010703 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.010678 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 20:21:38.010839 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.010682 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4qz5l\"" Jul 30 20:21:38.130433 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.130396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:38.131153 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.130769 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 20:21:38.131153 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.130833 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls podName:6878b2de-cd48-439d-859d-38e28a60e621 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:40.130814416 +0000 UTC m=+38.595952758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-lfjkk" (UID: "6878b2de-cd48-439d-859d-38e28a60e621") : secret "samples-operator-tls" not found Jul 30 20:21:38.231302 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.231268 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:38.231453 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.231321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:38.231453 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.231381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:38.231453 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.231434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:38.231619 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.231465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:38.231691 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.231671 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:38.231747 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.231731 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls podName:277d11be-7c47-46e8-8d2a-5e4016c305a7 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:40.231712811 +0000 UTC m=+38.696851161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls") pod "dns-default-nvthr" (UID: "277d11be-7c47-46e8-8d2a-5e4016c305a7") : secret "dns-default-metrics-tls" not found Jul 30 20:21:38.232141 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.232125 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:40.232110574 +0000 UTC m=+38.697248904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : configmap references non-existent config key: service-ca.crt Jul 30 20:21:38.232279 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.232205 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 20:21:38.232279 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.232239 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:40.232227808 +0000 UTC m=+38.697366137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : secret "router-metrics-certs-default" not found Jul 30 20:21:38.232403 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.232294 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:38.232403 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.232325 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls podName:7ba3571c-c95e-4bc3-a238-eee309a1b456 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:40.232314337 +0000 UTC m=+38.697452667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-xqtfv" (UID: "7ba3571c-c95e-4bc3-a238-eee309a1b456") : secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:38.232403 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.232388 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 20:21:38.232403 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.232398 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5fff7f9-2fsh7: secret "image-registry-tls" not found Jul 30 20:21:38.232693 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:38.232427 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls podName:b78f9503-3ff3-4800-92ef-da6471ad6071 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:40.232417366 +0000 UTC m=+38.697555701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls") pod "image-registry-5bb5fff7f9-2fsh7" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071") : secret "image-registry-tls" not found Jul 30 20:21:38.238727 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.238701 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" event={"ID":"9c96510c-70a4-411f-952b-1a4176ab3094","Type":"ContainerStarted","Data":"e0155905ccfd6123470a62431a0e795edca5071e9832dff5c1204bf0e02e1d33"} Jul 30 20:21:38.241078 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.241014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxt4p" event={"ID":"fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685","Type":"ContainerStarted","Data":"87e501e14d713b99d8fd5e10956ab8a362db6dd35e36d9f73c63159da2e98f3f"} Jul 30 20:21:38.242779 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.242713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" event={"ID":"6cd73e85-ac41-4b79-aa4d-3fcb090e046e","Type":"ContainerStarted","Data":"1683c72f5a7c2d9cdf40e1c7dc2235d40b021d97c2921938c60f8e8c65a1b60e"} Jul 30 20:21:38.246730 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.246444 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj" event={"ID":"0f6c52e7-2883-4cc4-9eca-d526502011a5","Type":"ContainerStarted","Data":"8b3d885d163f0d8d75afbea3c296e76750e3da0675024e06d0f9994413d61a70"} Jul 30 20:21:38.252689 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.252668 2575 generic.go:356] "Generic (PLEG): container finished" podID="41464378-d96a-4890-bcdb-fb23717b7593" containerID="0ed39a8e45055edf514186d1428f4c7083743b7f414eec66ee0802db84f679b8" exitCode=0 Jul 30 20:21:38.252785 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.252739 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" event={"ID":"41464378-d96a-4890-bcdb-fb23717b7593","Type":"ContainerDied","Data":"0ed39a8e45055edf514186d1428f4c7083743b7f414eec66ee0802db84f679b8"} Jul 30 20:21:38.258798 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.258746 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx" event={"ID":"a8624b77-b71d-41ac-a096-ec848d11f772","Type":"ContainerStarted","Data":"b1173bdbfd6ca9131412f351498f2899dc94deeba6ff0f99cdadbe89f98bc5ca"} Jul 30 20:21:38.262317 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.262294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wbkxk" event={"ID":"84cde953-2d57-4ce5-8848-8079a5023664","Type":"ContainerStarted","Data":"29ac7807ae6cee2cb7f53f21bc7441c40ee1d4dab9a72a779e71bc2784fc8dbd"} Jul 30 20:21:38.266938 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:38.266873 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" event={"ID":"a0f10072-fc2c-4d23-91b3-1a982a9decbd","Type":"ContainerStarted","Data":"8b397effb4544d8d00b226d25844d6a5a62690511ea80c65c93ed1239e085061"} Jul 30 20:21:39.281120 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:39.281083 2575 generic.go:356] "Generic (PLEG): container finished" podID="41464378-d96a-4890-bcdb-fb23717b7593" containerID="803fa9ec50098389af3635c2d068d5ff7126d10b549922155459d91ab77cc4f9" exitCode=0 Jul 30 20:21:39.281763 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:39.281164 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" event={"ID":"41464378-d96a-4890-bcdb-fb23717b7593","Type":"ContainerDied","Data":"803fa9ec50098389af3635c2d068d5ff7126d10b549922155459d91ab77cc4f9"} Jul 30 20:21:40.156092 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:40.156025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:40.156309 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.156290 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 20:21:40.156389 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.156374 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls podName:6878b2de-cd48-439d-859d-38e28a60e621 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:44.156346191 +0000 UTC m=+42.621484521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-lfjkk" (UID: "6878b2de-cd48-439d-859d-38e28a60e621") : secret "samples-operator-tls" not found Jul 30 20:21:40.256598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:40.256560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:40.256771 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:40.256631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:40.256771 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:40.256678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:40.256771 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:40.256761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:40.256939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:40.256793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:40.256939 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.256918 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 20:21:40.257034 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.256975 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:44.256956516 +0000 UTC m=+42.722094846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : secret "router-metrics-certs-default" not found Jul 30 20:21:40.257737 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.257380 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:40.257737 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.257433 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls podName:7ba3571c-c95e-4bc3-a238-eee309a1b456 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:44.257417601 +0000 UTC m=+42.722555931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-xqtfv" (UID: "7ba3571c-c95e-4bc3-a238-eee309a1b456") : secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:40.257737 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.257497 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 20:21:40.257737 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.257507 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5fff7f9-2fsh7: secret "image-registry-tls" not found Jul 30 20:21:40.257737 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.257538 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls podName:b78f9503-3ff3-4800-92ef-da6471ad6071 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:44.257527679 +0000 UTC m=+42.722666022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls") pod "image-registry-5bb5fff7f9-2fsh7" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071") : secret "image-registry-tls" not found Jul 30 20:21:40.257737 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.257606 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:40.257737 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.257638 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls podName:277d11be-7c47-46e8-8d2a-5e4016c305a7 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:44.257627079 +0000 UTC m=+42.722765408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls") pod "dns-default-nvthr" (UID: "277d11be-7c47-46e8-8d2a-5e4016c305a7") : secret "dns-default-metrics-tls" not found Jul 30 20:21:40.257737 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:40.257708 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:44.257697892 +0000 UTC m=+42.722836224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : configmap references non-existent config key: service-ca.crt Jul 30 20:21:40.293087 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:40.291741 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" event={"ID":"41464378-d96a-4890-bcdb-fb23717b7593","Type":"ContainerStarted","Data":"858efd59f1b19ef622d118c50a9f43585b77c2d6fb109d55d21c520ab896911b"} Jul 30 20:21:40.323036 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:40.322757 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zkqt8" podStartSLOduration=9.006588059 podStartE2EDuration="38.322740796s" totalImagesPullingTime="29.316152737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:40.321562865 +0000 UTC m=+38.786701226" watchObservedRunningTime="2026-07-30 20:21:40.322740796 +0000 UTC m=+38.787879159" Jul 30 20:21:44.193833 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:44.193619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:44.194479 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.193779 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 20:21:44.194479 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.193960 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls podName:6878b2de-cd48-439d-859d-38e28a60e621 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.193938476 +0000 UTC m=+50.659076802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-lfjkk" (UID: "6878b2de-cd48-439d-859d-38e28a60e621") : secret "samples-operator-tls" not found Jul 30 20:21:44.294743 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:44.294710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:44.294892 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:44.294761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:44.294892 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.294860 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 20:21:44.294892 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.294866 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:44.295031 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:44.294911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:44.295031 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.294934 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls podName:7ba3571c-c95e-4bc3-a238-eee309a1b456 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.294915378 +0000 UTC m=+50.760053708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-xqtfv" (UID: "7ba3571c-c95e-4bc3-a238-eee309a1b456") : secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:44.295031 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.294874 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5fff7f9-2fsh7: secret "image-registry-tls" not found Jul 30 20:21:44.295031 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.294981 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:44.295031 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.295004 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls podName:b78f9503-3ff3-4800-92ef-da6471ad6071 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.294985394 +0000 UTC m=+50.760123775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls") pod "image-registry-5bb5fff7f9-2fsh7" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071") : secret "image-registry-tls" not found Jul 30 20:21:44.295031 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.295025 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls podName:277d11be-7c47-46e8-8d2a-5e4016c305a7 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.295015116 +0000 UTC m=+50.760153441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls") pod "dns-default-nvthr" (UID: "277d11be-7c47-46e8-8d2a-5e4016c305a7") : secret "dns-default-metrics-tls" not found Jul 30 20:21:44.295354 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:44.295095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:44.295354 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:44.295120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:44.295354 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.295208 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 20:21:44.295354 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.295225 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.29521236 +0000 UTC m=+50.760350693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : configmap references non-existent config key: service-ca.crt Jul 30 20:21:44.295354 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:44.295248 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.295237794 +0000 UTC m=+50.760376137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : secret "router-metrics-certs-default" not found Jul 30 20:21:45.308209 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:45.308171 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx" event={"ID":"a8624b77-b71d-41ac-a096-ec848d11f772","Type":"ContainerStarted","Data":"bc52d72f2d3af540736e732825bb13dc1ab44d9af8a314693c811044c88d632a"} Jul 30 20:21:45.310405 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:45.310325 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wbkxk" event={"ID":"84cde953-2d57-4ce5-8848-8079a5023664","Type":"ContainerStarted","Data":"2f89edd9bcc477a033e52d56153b2b7793b07902f27fdc5841f376e740d8b16b"} Jul 30 20:21:45.312494 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:45.312471 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj" event={"ID":"0f6c52e7-2883-4cc4-9eca-d526502011a5","Type":"ContainerStarted","Data":"030e2f08a4e08440b6f10e0eab590e3af90a276314feb422ba0428ab89f70e25"} Jul 30 20:21:45.330306 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:45.330216 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-wbkxk" podStartSLOduration=15.81405199 podStartE2EDuration="23.330201699s" totalImagesPullingTime="7.516149709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:45.329330211 +0000 UTC m=+43.794468564" watchObservedRunningTime="2026-07-30 20:21:45.330201699 +0000 UTC m=+43.795340024" Jul 30 20:21:45.368432 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:45.367817 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-242bj" podStartSLOduration=15.962996128 podStartE2EDuration="23.367793328s" totalImagesPullingTime="7.4047972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:45.366802221 +0000 UTC m=+43.831940570" watchObservedRunningTime="2026-07-30 20:21:45.367793328 +0000 UTC m=+43.832931678" Jul 30 20:21:46.320929 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.320894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxt4p" event={"ID":"fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685","Type":"ContainerStarted","Data":"0ea48e276e5f2b555492976091cca3295cbbc3bce271dc1d3615b5a899ab6142"} Jul 30 20:21:46.322742 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.322688 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/0.log" Jul 30 20:21:46.322742 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.322732 2575 generic.go:356] "Generic (PLEG): container finished" podID="6cd73e85-ac41-4b79-aa4d-3fcb090e046e" containerID="f56331dfab9bcec0b88cd3c1b9c5f16f4903d6b9f391589475d0c7a78a64ce1e" exitCode=255 Jul 30 20:21:46.322902 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.322819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" event={"ID":"6cd73e85-ac41-4b79-aa4d-3fcb090e046e","Type":"ContainerDied","Data":"f56331dfab9bcec0b88cd3c1b9c5f16f4903d6b9f391589475d0c7a78a64ce1e"} Jul 30 20:21:46.323004 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.322979 2575 scope.go:117] "RemoveContainer" containerID="f56331dfab9bcec0b88cd3c1b9c5f16f4903d6b9f391589475d0c7a78a64ce1e" Jul 30 20:21:46.326811 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.326787 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" event={"ID":"a0f10072-fc2c-4d23-91b3-1a982a9decbd","Type":"ContainerStarted","Data":"a11752fa210a2a853290ffa02bce7c84b4059d10cbba59a2978ecf06d3b4ccb8"} Jul 30 20:21:46.328647 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.328627 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" event={"ID":"9c96510c-70a4-411f-952b-1a4176ab3094","Type":"ContainerStarted","Data":"28a4160e12ab530e9439eb4f6756c91490941026168619c644637e33c140afe8"} Jul 30 20:21:46.340198 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.340124 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wxt4p" podStartSLOduration=2.83053621 podStartE2EDuration="10.340110511s" totalImagesPullingTime="7.509574301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:46.33894006 +0000 UTC m=+44.804078432" watchObservedRunningTime="2026-07-30 20:21:46.340110511 +0000 UTC m=+44.805248836" Jul 30 20:21:46.355726 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.355680 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-kshwx" podStartSLOduration=16.819220795 podStartE2EDuration="24.355662326s" totalImagesPullingTime="7.536441531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:46.35484617 +0000 UTC m=+44.819984530" watchObservedRunningTime="2026-07-30 20:21:46.355662326 +0000 UTC m=+44.820800675" Jul 30 20:21:46.401485 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.401438 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" podStartSLOduration=16.887427631 podStartE2EDuration="24.401419561s" totalImagesPullingTime="7.51399193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:46.399589276 +0000 UTC m=+44.864727627" watchObservedRunningTime="2026-07-30 20:21:46.401419561 +0000 UTC m=+44.866557911" Jul 30 20:21:46.520617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.520569 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" podStartSLOduration=17.00312979 podStartE2EDuration="24.520548256s" totalImagesPullingTime="7.517418466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:46.421029969 +0000 UTC m=+44.886168321" watchObservedRunningTime="2026-07-30 20:21:46.520548256 +0000 UTC m=+44.985686606" Jul 30 20:21:46.521227 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.521208 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc"] Jul 30 20:21:46.534551 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.534532 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" Jul 30 20:21:46.537711 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.537678 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 30 20:21:46.537846 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.537678 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:46.537972 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.537827 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w5zh2\"" Jul 30 20:21:46.539581 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.539553 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc"] Jul 30 20:21:46.616450 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.616428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wwk\" (UniqueName: \"kubernetes.io/projected/6c6076ec-c274-4e50-ad5c-8dbbfc677104-kube-api-access-q2wwk\") pod \"migrator-68bc745d87-jcgbc\" (UID: \"6c6076ec-c274-4e50-ad5c-8dbbfc677104\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" Jul 30 20:21:46.717844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.717808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wwk\" (UniqueName: \"kubernetes.io/projected/6c6076ec-c274-4e50-ad5c-8dbbfc677104-kube-api-access-q2wwk\") pod \"migrator-68bc745d87-jcgbc\" (UID: \"6c6076ec-c274-4e50-ad5c-8dbbfc677104\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" Jul 30 20:21:46.729529 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.729432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wwk\" (UniqueName: \"kubernetes.io/projected/6c6076ec-c274-4e50-ad5c-8dbbfc677104-kube-api-access-q2wwk\") pod \"migrator-68bc745d87-jcgbc\" (UID: \"6c6076ec-c274-4e50-ad5c-8dbbfc677104\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" Jul 30 20:21:46.811257 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.811226 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:46.811383 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.811272 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:46.857526 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:46.857502 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" Jul 30 20:21:47.009757 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.009683 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc"] Jul 30 20:21:47.012717 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:21:47.012687 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6076ec_c274_4e50_ad5c_8dbbfc677104.slice/crio-399cd8c9eeb750bec5b0cdaa5043cef201e5f1acba2cd30464918e0b466a15f7 WatchSource:0}: Error finding container 399cd8c9eeb750bec5b0cdaa5043cef201e5f1acba2cd30464918e0b466a15f7: Status 404 returned error can't find the container with id 399cd8c9eeb750bec5b0cdaa5043cef201e5f1acba2cd30464918e0b466a15f7 Jul 30 20:21:47.065629 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.065599 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f8s8v"] Jul 30 20:21:47.098390 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.098362 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8s8v"] Jul 30 20:21:47.098523 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.098505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.101563 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.101538 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 20:21:47.101747 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.101728 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9q9km\"" Jul 30 20:21:47.101881 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.101866 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 20:21:47.221882 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.221854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjjx\" (UniqueName: \"kubernetes.io/projected/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-api-access-ndjjx\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.221991 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.221906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-crio-socket\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.221991 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.221924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.221991 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.221948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-data-volume\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.222152 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.222135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.322761 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.322687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.322761 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.322728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjjx\" (UniqueName: \"kubernetes.io/projected/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-api-access-ndjjx\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.322781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-crio-socket\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.322806 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.322809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.322873 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:21:47.822857819 +0000 UTC m=+46.287996146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : configmap "kube-rbac-proxy" not found Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.322909 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.322963 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:21:47.822946472 +0000 UTC m=+46.288084798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.322910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-data-volume\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.323003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-crio-socket\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.323204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.323175 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-data-volume\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.334473 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.334449 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/1.log" Jul 30 20:21:47.334864 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.334847 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/0.log" Jul 30 20:21:47.334963 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.334880 2575 generic.go:356] "Generic (PLEG): container finished" podID="6cd73e85-ac41-4b79-aa4d-3fcb090e046e" containerID="7d7e5ff8d29626f109d1cf86ce064762260797b53043354b769595a9b9c328c1" exitCode=255 Jul 30 20:21:47.334963 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.334911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" event={"ID":"6cd73e85-ac41-4b79-aa4d-3fcb090e046e","Type":"ContainerDied","Data":"7d7e5ff8d29626f109d1cf86ce064762260797b53043354b769595a9b9c328c1"} Jul 30 20:21:47.335083 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.334972 2575 scope.go:117] "RemoveContainer" containerID="f56331dfab9bcec0b88cd3c1b9c5f16f4903d6b9f391589475d0c7a78a64ce1e" Jul 30 20:21:47.335187 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.335169 2575 scope.go:117] "RemoveContainer" containerID="7d7e5ff8d29626f109d1cf86ce064762260797b53043354b769595a9b9c328c1" Jul 30 20:21:47.335371 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.335347 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4lfdb_openshift-console-operator(6cd73e85-ac41-4b79-aa4d-3fcb090e046e)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" podUID="6cd73e85-ac41-4b79-aa4d-3fcb090e046e" Jul 30 20:21:47.335934 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.335909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjjx\" (UniqueName: \"kubernetes.io/projected/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-api-access-ndjjx\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.336408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.336230 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" event={"ID":"6c6076ec-c274-4e50-ad5c-8dbbfc677104","Type":"ContainerStarted","Data":"399cd8c9eeb750bec5b0cdaa5043cef201e5f1acba2cd30464918e0b466a15f7"} Jul 30 20:21:47.827901 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.827863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.828078 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:47.827943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:47.828078 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.827998 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 20:21:47.828188 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.828081 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:47.828188 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.828100 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:21:48.828083349 +0000 UTC m=+47.293221676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : configmap "kube-rbac-proxy" not found Jul 30 20:21:47.828188 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:47.828148 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:21:48.828130331 +0000 UTC m=+47.293268661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:48.341569 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:48.341540 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/1.log" Jul 30 20:21:48.342041 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:48.341865 2575 scope.go:117] "RemoveContainer" containerID="7d7e5ff8d29626f109d1cf86ce064762260797b53043354b769595a9b9c328c1" Jul 30 20:21:48.342505 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:48.342465 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4lfdb_openshift-console-operator(6cd73e85-ac41-4b79-aa4d-3fcb090e046e)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" podUID="6cd73e85-ac41-4b79-aa4d-3fcb090e046e" Jul 30 20:21:48.838392 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:48.838364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:48.838487 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:48.838458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:48.838487 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:48.838471 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 20:21:48.838549 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:48.838523 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:21:50.838504875 +0000 UTC m=+49.303643202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : configmap "kube-rbac-proxy" not found Jul 30 20:21:48.838609 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:48.838596 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:48.838665 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:48.838655 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:21:50.838637778 +0000 UTC m=+49.303776119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:49.346434 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:49.346397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" event={"ID":"6c6076ec-c274-4e50-ad5c-8dbbfc677104","Type":"ContainerStarted","Data":"547d9da7aca27d158355f30a509fec74dc751e5ad02cec37954460c865b26cb4"} Jul 30 20:21:49.346796 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:49.346478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" event={"ID":"6c6076ec-c274-4e50-ad5c-8dbbfc677104","Type":"ContainerStarted","Data":"717d4d8bcf8143ac03b0d8f3a1d5b5ab35cf92d562b4ef0e4f933498912a9492"} Jul 30 20:21:49.364306 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:49.364251 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-jcgbc" podStartSLOduration=1.742636072 podStartE2EDuration="3.364233431s" totalImagesPullingTime="1.621597359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:49.363985616 +0000 UTC m=+47.829123990" watchObservedRunningTime="2026-07-30 20:21:49.364233431 +0000 UTC m=+47.829371780" Jul 30 20:21:49.645764 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:49.645705 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mfzp_ff93e08a-42e5-457b-96c2-a54192e5518b/dns-node-resolver/0.log" Jul 30 20:21:50.442842 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:50.442812 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ktn4b_da966ad2-6153-4de3-8d61-4f80ce2fda88/node-ca/0.log" Jul 30 20:21:50.851319 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:50.851293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:50.851493 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:50.851346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:50.851493 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:50.851420 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 20:21:50.851493 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:50.851438 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:50.851622 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:50.851493 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:21:54.851474906 +0000 UTC m=+53.316613251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : configmap "kube-rbac-proxy" not found Jul 30 20:21:50.851622 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:50.851526 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:21:54.851506892 +0000 UTC m=+53.316645232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:51.237830 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:51.237805 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxt4p_fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685/serve-healthcheck-canary/0.log" Jul 30 20:21:51.437474 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:51.437449 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-jcgbc_6c6076ec-c274-4e50-ad5c-8dbbfc677104/migrator/0.log" Jul 30 20:21:51.637554 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:51.637486 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-jcgbc_6c6076ec-c274-4e50-ad5c-8dbbfc677104/graceful-termination/0.log" Jul 30 20:21:51.839302 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:51.839273 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-xx7xc_a0f10072-fc2c-4d23-91b3-1a982a9decbd/kube-storage-version-migrator-operator/0.log" Jul 30 20:21:52.262887 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:52.262853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:21:52.263040 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.263012 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 20:21:52.263106 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.263084 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls podName:6878b2de-cd48-439d-859d-38e28a60e621 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:08.263068934 +0000 UTC m=+66.728207260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-lfjkk" (UID: "6878b2de-cd48-439d-859d-38e28a60e621") : secret "samples-operator-tls" not found Jul 30 20:21:52.363511 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:52.363485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:52.363641 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:52.363520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:21:52.363641 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:52.363554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:21:52.363641 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:52.363589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:21:52.363641 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:52.363607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363644 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363655 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:22:08.363633824 +0000 UTC m=+66.828772150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : configmap references non-existent config key: service-ca.crt Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363688 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363725 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls podName:277d11be-7c47-46e8-8d2a-5e4016c305a7 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:08.363714419 +0000 UTC m=+66.828852745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls") pod "dns-default-nvthr" (UID: "277d11be-7c47-46e8-8d2a-5e4016c305a7") : secret "dns-default-metrics-tls" not found Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363740 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363766 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs podName:7c5e8c92-080a-4b16-844a-e3310c5dc43f nodeName:}" failed. No retries permitted until 2026-07-30 20:22:08.36373394 +0000 UTC m=+66.828872277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs") pod "router-default-6c89bb85d8-xz8kr" (UID: "7c5e8c92-080a-4b16-844a-e3310c5dc43f") : secret "router-metrics-certs-default" not found Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363771 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363789 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls podName:7ba3571c-c95e-4bc3-a238-eee309a1b456 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:08.363774051 +0000 UTC m=+66.828912389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-xqtfv" (UID: "7ba3571c-c95e-4bc3-a238-eee309a1b456") : secret "cluster-monitoring-operator-tls" not found Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363790 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb5fff7f9-2fsh7: secret "image-registry-tls" not found Jul 30 20:21:52.363845 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:52.363851 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls podName:b78f9503-3ff3-4800-92ef-da6471ad6071 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:08.363835107 +0000 UTC m=+66.828973447 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls") pod "image-registry-5bb5fff7f9-2fsh7" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071") : secret "image-registry-tls" not found Jul 30 20:21:54.885991 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:54.885960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:54.886386 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:54.886090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:54.886386 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:54.886137 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:54.886386 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:54.886207 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls podName:d0ff4c71-e236-4d4b-aab7-22eea0229a1b nodeName:}" failed. No retries permitted until 2026-07-30 20:22:02.886191903 +0000 UTC m=+61.351330238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s8v" (UID: "d0ff4c71-e236-4d4b-aab7-22eea0229a1b") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:54.886634 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:54.886618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:21:56.811877 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:56.811828 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:56.811877 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:56.811896 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:21:56.812466 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:56.812238 2575 scope.go:117] "RemoveContainer" containerID="7d7e5ff8d29626f109d1cf86ce064762260797b53043354b769595a9b9c328c1" Jul 30 20:21:57.373852 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:57.373829 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:21:57.374234 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:57.374220 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/1.log" Jul 30 20:21:57.374295 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:57.374251 2575 generic.go:356] "Generic (PLEG): container finished" podID="6cd73e85-ac41-4b79-aa4d-3fcb090e046e" containerID="0181f14109722971c98ed9848abd80ad0e06799f1c60672235dc34f5bfe98628" exitCode=255 Jul 30 20:21:57.374331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:57.374311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" event={"ID":"6cd73e85-ac41-4b79-aa4d-3fcb090e046e","Type":"ContainerDied","Data":"0181f14109722971c98ed9848abd80ad0e06799f1c60672235dc34f5bfe98628"} Jul 30 20:21:57.374369 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:57.374357 2575 scope.go:117] "RemoveContainer" containerID="7d7e5ff8d29626f109d1cf86ce064762260797b53043354b769595a9b9c328c1" Jul 30 20:21:57.374649 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:57.374631 2575 scope.go:117] "RemoveContainer" containerID="0181f14109722971c98ed9848abd80ad0e06799f1c60672235dc34f5bfe98628" Jul 30 20:21:57.374837 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:21:57.374814 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4lfdb_openshift-console-operator(6cd73e85-ac41-4b79-aa4d-3fcb090e046e)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" podUID="6cd73e85-ac41-4b79-aa4d-3fcb090e046e" Jul 30 20:21:58.378493 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:21:58.378464 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:22:02.240498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:02.240464 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99c7" Jul 30 20:22:02.957254 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:02.957197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:22:02.959513 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:02.959478 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0ff4c71-e236-4d4b-aab7-22eea0229a1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s8v\" (UID: \"d0ff4c71-e236-4d4b-aab7-22eea0229a1b\") " pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:22:03.011435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:03.011405 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9q9km\"" Jul 30 20:22:03.019687 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:03.019662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8s8v" Jul 30 20:22:03.146512 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:03.146480 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8s8v"] Jul 30 20:22:03.149442 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:03.149411 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ff4c71_e236_4d4b_aab7_22eea0229a1b.slice/crio-bf1a0c2a002ab490f6965e083c10c8c65a6ca4c2cc2736572b0c95d6360948f0 WatchSource:0}: Error finding container bf1a0c2a002ab490f6965e083c10c8c65a6ca4c2cc2736572b0c95d6360948f0: Status 404 returned error can't find the container with id bf1a0c2a002ab490f6965e083c10c8c65a6ca4c2cc2736572b0c95d6360948f0 Jul 30 20:22:03.394319 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:03.394233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8s8v" event={"ID":"d0ff4c71-e236-4d4b-aab7-22eea0229a1b","Type":"ContainerStarted","Data":"2040f055b25186889a7f128f98a8c6a72c1658fdbb4ece98262fabf3499cccf8"} Jul 30 20:22:03.394319 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:03.394296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8s8v" event={"ID":"d0ff4c71-e236-4d4b-aab7-22eea0229a1b","Type":"ContainerStarted","Data":"bf1a0c2a002ab490f6965e083c10c8c65a6ca4c2cc2736572b0c95d6360948f0"} Jul 30 20:22:05.403527 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:05.403491 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8s8v" event={"ID":"d0ff4c71-e236-4d4b-aab7-22eea0229a1b","Type":"ContainerStarted","Data":"86d69ae69035d8b452949244b0d78a3d89d98e25c7691cc5c8009a889c7ef7b6"} Jul 30 20:22:06.412412 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:06.408767 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8s8v" event={"ID":"d0ff4c71-e236-4d4b-aab7-22eea0229a1b","Type":"ContainerStarted","Data":"4c3c000fb387d2c1332f384ea734f834325a9e276072b542eaeac08fe4451a3e"} Jul 30 20:22:06.430925 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:06.430873 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f8s8v" podStartSLOduration=16.875035799 podStartE2EDuration="19.430857737s" totalImagesPullingTime="2.555821938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:06.428929514 +0000 UTC m=+64.894067862" watchObservedRunningTime="2026-07-30 20:22:06.430857737 +0000 UTC m=+64.895996084" Jul 30 20:22:06.811028 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:06.811000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:22:06.811187 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:06.811091 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:22:06.811387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:06.811374 2575 scope.go:117] "RemoveContainer" containerID="0181f14109722971c98ed9848abd80ad0e06799f1c60672235dc34f5bfe98628" Jul 30 20:22:06.811528 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:22:06.811511 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4lfdb_openshift-console-operator(6cd73e85-ac41-4b79-aa4d-3fcb090e046e)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" podUID="6cd73e85-ac41-4b79-aa4d-3fcb090e046e" Jul 30 20:22:07.904521 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:07.904477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:22:07.904992 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:07.904549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:22:07.907573 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:07.907551 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 20:22:07.908660 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:07.908644 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 20:22:07.917243 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:07.917210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19e1b2f5-58e3-423b-80d7-51670c463768-original-pull-secret\") pod \"global-pull-secret-syncer-6htjd\" (UID: \"19e1b2f5-58e3-423b-80d7-51670c463768\") " pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:22:07.917243 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:07.917237 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc8c488-8fee-4f10-a041-8d67bc0569a8-metrics-certs\") pod \"network-metrics-daemon-dgsbq\" (UID: \"dfc8c488-8fee-4f10-a041-8d67bc0569a8\") " pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:22:07.920122 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:07.920100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6htjd" Jul 30 20:22:08.005701 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.005653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:22:08.008404 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.008378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79c4s\" (UniqueName: \"kubernetes.io/projected/27a0cc53-05f6-454e-9feb-28dd6673da84-kube-api-access-79c4s\") pod \"network-check-target-k8266\" (UID: \"27a0cc53-05f6-454e-9feb-28dd6673da84\") " pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:22:08.021173 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.021147 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4qz5l\"" Jul 30 20:22:08.029454 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.029427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dgsbq" Jul 30 20:22:08.047013 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.046987 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6htjd"] Jul 30 20:22:08.049762 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:08.049733 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e1b2f5_58e3_423b_80d7_51670c463768.slice/crio-0c37ec69606ea768e331dfa0d0103819a33561a6e029b5279581942df926454a WatchSource:0}: Error finding container 0c37ec69606ea768e331dfa0d0103819a33561a6e029b5279581942df926454a: Status 404 returned error can't find the container with id 0c37ec69606ea768e331dfa0d0103819a33561a6e029b5279581942df926454a Jul 30 20:22:08.153088 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.153038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dgsbq"] Jul 30 20:22:08.155235 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:08.155187 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc8c488_8fee_4f10_a041_8d67bc0569a8.slice/crio-76e8a6199e27794f9ab5bbeea0d632db13eea1fb4a17ac70e3678292aa112df8 WatchSource:0}: Error finding container 76e8a6199e27794f9ab5bbeea0d632db13eea1fb4a17ac70e3678292aa112df8: Status 404 returned error can't find the container with id 76e8a6199e27794f9ab5bbeea0d632db13eea1fb4a17ac70e3678292aa112df8 Jul 30 20:22:08.229474 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.229450 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jh8wl\"" Jul 30 20:22:08.236724 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.236701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:22:08.308517 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.308490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:22:08.310707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.310682 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6878b2de-cd48-439d-859d-38e28a60e621-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-lfjkk\" (UID: \"6878b2de-cd48-439d-859d-38e28a60e621\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:22:08.403399 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.403374 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k8266"] Jul 30 20:22:08.405859 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:08.405799 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a0cc53_05f6_454e_9feb_28dd6673da84.slice/crio-10a7df7f1b24176ec8a5863fa813b7d79fd24f89cdd2162fe15c58fa5f4608bf WatchSource:0}: Error finding container 10a7df7f1b24176ec8a5863fa813b7d79fd24f89cdd2162fe15c58fa5f4608bf: Status 404 returned error can't find the container with id 10a7df7f1b24176ec8a5863fa813b7d79fd24f89cdd2162fe15c58fa5f4608bf Jul 30 20:22:08.409717 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.409694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:08.409799 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.409728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:08.409799 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.409757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:22:08.409872 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.409798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:22:08.409872 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.409824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:22:08.410451 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.410418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5e8c92-080a-4b16-844a-e3310c5dc43f-service-ca-bundle\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:08.412170 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.412126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277d11be-7c47-46e8-8d2a-5e4016c305a7-metrics-tls\") pod \"dns-default-nvthr\" (UID: \"277d11be-7c47-46e8-8d2a-5e4016c305a7\") " pod="openshift-dns/dns-default-nvthr" Jul 30 20:22:08.412263 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.412221 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ba3571c-c95e-4bc3-a238-eee309a1b456-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-xqtfv\" (UID: \"7ba3571c-c95e-4bc3-a238-eee309a1b456\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:22:08.412263 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.412219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"image-registry-5bb5fff7f9-2fsh7\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:22:08.412263 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.412247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5e8c92-080a-4b16-844a-e3310c5dc43f-metrics-certs\") pod \"router-default-6c89bb85d8-xz8kr\" (UID: \"7c5e8c92-080a-4b16-844a-e3310c5dc43f\") " pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:08.416035 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.416005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6htjd" event={"ID":"19e1b2f5-58e3-423b-80d7-51670c463768","Type":"ContainerStarted","Data":"0c37ec69606ea768e331dfa0d0103819a33561a6e029b5279581942df926454a"} Jul 30 20:22:08.416964 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.416939 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dgsbq" event={"ID":"dfc8c488-8fee-4f10-a041-8d67bc0569a8","Type":"ContainerStarted","Data":"76e8a6199e27794f9ab5bbeea0d632db13eea1fb4a17ac70e3678292aa112df8"} Jul 30 20:22:08.417916 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.417900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k8266" event={"ID":"27a0cc53-05f6-454e-9feb-28dd6673da84","Type":"ContainerStarted","Data":"10a7df7f1b24176ec8a5863fa813b7d79fd24f89cdd2162fe15c58fa5f4608bf"} Jul 30 20:22:08.541036 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.541005 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ct4r5\"" Jul 30 20:22:08.549176 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.549147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" Jul 30 20:22:08.559948 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.559912 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xkxnt\"" Jul 30 20:22:08.568192 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.568171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" Jul 30 20:22:08.571732 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.571701 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2nln4\"" Jul 30 20:22:08.579840 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.579769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:08.596914 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.596860 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q4l4z\"" Jul 30 20:22:08.604522 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.604449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:22:08.632638 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.632376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nmld7\"" Jul 30 20:22:08.640034 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.639963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nvthr" Jul 30 20:22:08.760969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.760117 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk"] Jul 30 20:22:08.793337 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.790236 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv"] Jul 30 20:22:08.840781 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.840542 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c89bb85d8-xz8kr"] Jul 30 20:22:08.844736 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:08.844651 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5e8c92_080a_4b16_844a_e3310c5dc43f.slice/crio-8e0fd02cef0187927d034a4d598337bb3caadd055193736c1641875e02b80a3d WatchSource:0}: Error finding container 8e0fd02cef0187927d034a4d598337bb3caadd055193736c1641875e02b80a3d: Status 404 returned error can't find the container with id 8e0fd02cef0187927d034a4d598337bb3caadd055193736c1641875e02b80a3d Jul 30 20:22:08.861943 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.860420 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb5fff7f9-2fsh7"] Jul 30 20:22:08.865313 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:08.865285 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78f9503_3ff3_4800_92ef_da6471ad6071.slice/crio-bc148c63d1cfad9353a34d5036c47da6c6a0d1154f8d04647413ad98db51b75d WatchSource:0}: Error finding container bc148c63d1cfad9353a34d5036c47da6c6a0d1154f8d04647413ad98db51b75d: Status 404 returned error can't find the container with id bc148c63d1cfad9353a34d5036c47da6c6a0d1154f8d04647413ad98db51b75d Jul 30 20:22:08.881643 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:08.881612 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nvthr"] Jul 30 20:22:08.885514 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:08.885481 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277d11be_7c47_46e8_8d2a_5e4016c305a7.slice/crio-e7e51d4b80235ade904ebe87f08607cb0fa081d07a71738d7510faccafc1321b WatchSource:0}: Error finding container e7e51d4b80235ade904ebe87f08607cb0fa081d07a71738d7510faccafc1321b: Status 404 returned error can't find the container with id e7e51d4b80235ade904ebe87f08607cb0fa081d07a71738d7510faccafc1321b Jul 30 20:22:09.072375 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.072340 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bb5fff7f9-2fsh7"] Jul 30 20:22:09.432813 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.432775 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k8266" event={"ID":"27a0cc53-05f6-454e-9feb-28dd6673da84","Type":"ContainerStarted","Data":"1cbb263ea39dc50224f8ff26e28f84fe25c20219eb39d66f0c95b823f8d93194"} Jul 30 20:22:09.433030 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.432996 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:22:09.434614 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.434579 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nvthr" event={"ID":"277d11be-7c47-46e8-8d2a-5e4016c305a7","Type":"ContainerStarted","Data":"e7e51d4b80235ade904ebe87f08607cb0fa081d07a71738d7510faccafc1321b"} Jul 30 20:22:09.436118 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.436086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" event={"ID":"7ba3571c-c95e-4bc3-a238-eee309a1b456","Type":"ContainerStarted","Data":"80f32766c191ea39ccf30fd698c64c78355e3dbfad583518573e1d7936a18bca"} Jul 30 20:22:09.438605 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.438578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" event={"ID":"6878b2de-cd48-439d-859d-38e28a60e621","Type":"ContainerStarted","Data":"6235b6e4934579ba714772b83965a238011d4654d43153f8be324ba4c25f93aa"} Jul 30 20:22:09.440928 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.440899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" event={"ID":"b78f9503-3ff3-4800-92ef-da6471ad6071","Type":"ContainerStarted","Data":"21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757"} Jul 30 20:22:09.441027 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.440952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" event={"ID":"b78f9503-3ff3-4800-92ef-da6471ad6071","Type":"ContainerStarted","Data":"bc148c63d1cfad9353a34d5036c47da6c6a0d1154f8d04647413ad98db51b75d"} Jul 30 20:22:09.441027 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.441012 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:22:09.443537 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.443461 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" event={"ID":"7c5e8c92-080a-4b16-844a-e3310c5dc43f","Type":"ContainerStarted","Data":"9e6b699e6287a26dd0a8d5f1543f27b3024536d0d488cd7098f560d6d781746f"} Jul 30 20:22:09.443537 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.443508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" event={"ID":"7c5e8c92-080a-4b16-844a-e3310c5dc43f","Type":"ContainerStarted","Data":"8e0fd02cef0187927d034a4d598337bb3caadd055193736c1641875e02b80a3d"} Jul 30 20:22:09.450729 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.450674 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k8266" podStartSLOduration=67.450654758 podStartE2EDuration="1m7.450654758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:09.44987082 +0000 UTC m=+67.915009191" watchObservedRunningTime="2026-07-30 20:22:09.450654758 +0000 UTC m=+67.915793083" Jul 30 20:22:09.473431 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.473368 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" podStartSLOduration=47.473348744 podStartE2EDuration="47.473348744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:09.472587452 +0000 UTC m=+67.937725802" watchObservedRunningTime="2026-07-30 20:22:09.473348744 +0000 UTC m=+67.938487070" Jul 30 20:22:09.501244 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.501183 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" podStartSLOduration=47.501158766 podStartE2EDuration="47.501158766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:09.499520003 +0000 UTC m=+67.964658375" watchObservedRunningTime="2026-07-30 20:22:09.501158766 +0000 UTC m=+67.966297115" Jul 30 20:22:09.580245 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.580205 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:09.583171 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:09.583141 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:10.453978 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:10.453934 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dgsbq" event={"ID":"dfc8c488-8fee-4f10-a041-8d67bc0569a8","Type":"ContainerStarted","Data":"77205ca5d713373330e5a707c0fce80dcc3fecdc76a498f1f176cfa83d38a948"} Jul 30 20:22:10.454474 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:10.453996 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dgsbq" event={"ID":"dfc8c488-8fee-4f10-a041-8d67bc0569a8","Type":"ContainerStarted","Data":"2d5e7ab0c0ff2fdb9cab2b243aaf7fdec9b24830cece1a97d7e312f56e82b166"} Jul 30 20:22:10.455163 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:10.455111 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:10.456666 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:10.456643 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c89bb85d8-xz8kr" Jul 30 20:22:10.470996 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:10.470937 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dgsbq" podStartSLOduration=66.737559176 podStartE2EDuration="1m8.470919422s" totalImagesPullingTime="1.733360246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:10.470048797 +0000 UTC m=+68.935187147" watchObservedRunningTime="2026-07-30 20:22:10.470919422 +0000 UTC m=+68.936057769" Jul 30 20:22:15.475987 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.475949 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nvthr" event={"ID":"277d11be-7c47-46e8-8d2a-5e4016c305a7","Type":"ContainerStarted","Data":"322b3c95fd9aceb1f01039d00311a8b6c06b9a9aae605a5f18810160f0dd65c3"} Jul 30 20:22:15.476522 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.476006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nvthr" event={"ID":"277d11be-7c47-46e8-8d2a-5e4016c305a7","Type":"ContainerStarted","Data":"6fa5e022167ebf00af524ff8cad475717c1158410dec876c64ed7af2fc4415f8"} Jul 30 20:22:15.476522 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.476088 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nvthr" Jul 30 20:22:15.477822 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.477794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" event={"ID":"7ba3571c-c95e-4bc3-a238-eee309a1b456","Type":"ContainerStarted","Data":"ddce2a11a7b102f75876bf0e2713c39eb2f580c536f1c221ae6c5cdcb1434640"} Jul 30 20:22:15.479210 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.479188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6htjd" event={"ID":"19e1b2f5-58e3-423b-80d7-51670c463768","Type":"ContainerStarted","Data":"a1ff71e4fa5eb514a5d40bede92afefaa4ed97c13857283a10fb74f242ecc34e"} Jul 30 20:22:15.480920 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.480890 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" event={"ID":"6878b2de-cd48-439d-859d-38e28a60e621","Type":"ContainerStarted","Data":"d791c3ec35ac0ae68ff0787a64bf51b4cd7321a4adfbbc3ab186d7c434599f03"} Jul 30 20:22:15.481028 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.480926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" event={"ID":"6878b2de-cd48-439d-859d-38e28a60e621","Type":"ContainerStarted","Data":"8c8d469ff7a48e7ee9312b1a138cebdf44dff1adece1e09752076fdb05cd135d"} Jul 30 20:22:15.499281 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.499225 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nvthr" podStartSLOduration=33.742940999 podStartE2EDuration="39.499206289s" totalImagesPullingTime="5.75626529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:15.497014827 +0000 UTC m=+73.962153176" watchObservedRunningTime="2026-07-30 20:22:15.499206289 +0000 UTC m=+73.964344639" Jul 30 20:22:15.513271 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.513207 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6htjd" podStartSLOduration=66.919836718 podStartE2EDuration="1m13.513186519s" totalImagesPullingTime="6.593349801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:15.512014121 +0000 UTC m=+73.977152471" watchObservedRunningTime="2026-07-30 20:22:15.513186519 +0000 UTC m=+73.978324871" Jul 30 20:22:15.552449 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.552389 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-lfjkk" podStartSLOduration=47.756615512 podStartE2EDuration="53.552368162s" totalImagesPullingTime="5.79575265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:15.531658742 +0000 UTC m=+73.996797094" watchObservedRunningTime="2026-07-30 20:22:15.552368162 +0000 UTC m=+74.017506549" Jul 30 20:22:15.552972 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:15.552944 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-xqtfv" podStartSLOduration=47.713736803 podStartE2EDuration="53.552935053s" totalImagesPullingTime="5.83919825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:15.551876509 +0000 UTC m=+74.017014869" watchObservedRunningTime="2026-07-30 20:22:15.552935053 +0000 UTC m=+74.018073404" Jul 30 20:22:19.007078 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:19.007026 2575 scope.go:117] "RemoveContainer" containerID="0181f14109722971c98ed9848abd80ad0e06799f1c60672235dc34f5bfe98628" Jul 30 20:22:19.498095 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:19.498048 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:22:19.498286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:19.498183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" event={"ID":"6cd73e85-ac41-4b79-aa4d-3fcb090e046e","Type":"ContainerStarted","Data":"4e7891dec39e5dccd31573ff31ff9e5b22ffd6de37d64137b03a0bc1acf1f676"} Jul 30 20:22:19.498489 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:19.498469 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:22:19.516763 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:19.516709 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" podStartSLOduration=50.001037439 podStartE2EDuration="57.516691701s" totalImagesPullingTime="7.515654262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:19.516349185 +0000 UTC m=+77.981487532" watchObservedRunningTime="2026-07-30 20:22:19.516691701 +0000 UTC m=+77.981830049" Jul 30 20:22:20.215432 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:20.215404 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4lfdb" Jul 30 20:22:25.486876 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.486750 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nvthr" Jul 30 20:22:25.774173 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.774090 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qm7rr"] Jul 30 20:22:25.778811 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.778791 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.781873 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.781849 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lm9q\"" Jul 30 20:22:25.781998 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.781935 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 20:22:25.782138 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.782117 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 20:22:25.782186 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.782134 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 20:22:25.782186 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.782163 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 20:22:25.954221 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a798888-f74f-444b-815a-1701950a86b9-metrics-client-ca\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.954381 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.954381 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-root\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.954381 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-wtmp\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.954381 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-tls\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.954563 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-sys\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.954563 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-textfile\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.954563 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:25.954684 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:25.954576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnq8\" (UniqueName: \"kubernetes.io/projected/4a798888-f74f-444b-815a-1701950a86b9-kube-api-access-9nnq8\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.054984 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.054906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnq8\" (UniqueName: \"kubernetes.io/projected/4a798888-f74f-444b-815a-1701950a86b9-kube-api-access-9nnq8\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.054984 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.054944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a798888-f74f-444b-815a-1701950a86b9-metrics-client-ca\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.054984 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.054974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055219 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-root\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055219 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055077 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-root\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055219 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-wtmp\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055337 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-tls\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055337 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-sys\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055337 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-textfile\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055337 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055509 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055341 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-wtmp\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055509 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a798888-f74f-444b-815a-1701950a86b9-sys\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055509 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:22:26.055406 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 30 20:22:26.055509 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:22:26.055495 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-tls podName:4a798888-f74f-444b-815a-1701950a86b9 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:26.555476155 +0000 UTC m=+85.020614482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-tls") pod "node-exporter-qm7rr" (UID: "4a798888-f74f-444b-815a-1701950a86b9") : secret "node-exporter-tls" not found Jul 30 20:22:26.055660 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055561 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a798888-f74f-444b-815a-1701950a86b9-metrics-client-ca\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055660 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055625 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-textfile\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.055809 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.055792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.057236 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.057216 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.064280 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.064255 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnq8\" (UniqueName: \"kubernetes.io/projected/4a798888-f74f-444b-815a-1701950a86b9-kube-api-access-9nnq8\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.560429 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.560383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-tls\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.562848 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.562822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a798888-f74f-444b-815a-1701950a86b9-node-exporter-tls\") pod \"node-exporter-qm7rr\" (UID: \"4a798888-f74f-444b-815a-1701950a86b9\") " pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.692435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:26.692396 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qm7rr" Jul 30 20:22:26.706869 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:26.706840 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a798888_f74f_444b_815a_1701950a86b9.slice/crio-e2566403a48a16c3a47612764d9932c65fcae75b2d810939627bdcfc3af6ba50 WatchSource:0}: Error finding container e2566403a48a16c3a47612764d9932c65fcae75b2d810939627bdcfc3af6ba50: Status 404 returned error can't find the container with id e2566403a48a16c3a47612764d9932c65fcae75b2d810939627bdcfc3af6ba50 Jul 30 20:22:27.530645 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:27.530607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm7rr" event={"ID":"4a798888-f74f-444b-815a-1701950a86b9","Type":"ContainerStarted","Data":"e2566403a48a16c3a47612764d9932c65fcae75b2d810939627bdcfc3af6ba50"} Jul 30 20:22:28.535892 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:28.535848 2575 generic.go:356] "Generic (PLEG): container finished" podID="4a798888-f74f-444b-815a-1701950a86b9" containerID="116070f55a4c59f29aaa0d92ea202e8de742f54597760dc20fd2eb8c6af3fc1b" exitCode=0 Jul 30 20:22:28.536346 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:28.535932 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm7rr" event={"ID":"4a798888-f74f-444b-815a-1701950a86b9","Type":"ContainerDied","Data":"116070f55a4c59f29aaa0d92ea202e8de742f54597760dc20fd2eb8c6af3fc1b"} Jul 30 20:22:29.542214 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:29.542178 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm7rr" event={"ID":"4a798888-f74f-444b-815a-1701950a86b9","Type":"ContainerStarted","Data":"915776248c13581ae7b3677d848aa5a9f8f614edb42d720d6c80845ed6f1391b"} Jul 30 20:22:29.542610 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:29.542233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm7rr" event={"ID":"4a798888-f74f-444b-815a-1701950a86b9","Type":"ContainerStarted","Data":"8af91912578376fe0ea0448e283760f4a1ada77f50eb0c818f3a3da404f30bfb"} Jul 30 20:22:29.562229 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:29.562176 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qm7rr" podStartSLOduration=3.639304328 podStartE2EDuration="4.56215812s" totalImagesPullingTime="857.061673ms" totalInitContainerRuntime="65.792119ms" isStatefulPod=true podCreationTimestamp="2026-07-30 20:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:29.560187538 +0000 UTC m=+88.025325902" watchObservedRunningTime="2026-07-30 20:22:29.56215812 +0000 UTC m=+88.027296467" Jul 30 20:22:30.459985 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.459948 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:22:30.498780 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.498740 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj"] Jul 30 20:22:30.502124 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.502105 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:30.504526 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.504502 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 30 20:22:30.504636 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.504533 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xx27c\"" Jul 30 20:22:30.510943 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.510907 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj"] Jul 30 20:22:30.590347 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.590311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ee38a94e-1190-4f5f-a4f3-2875d2eae79c-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vv4fj\" (UID: \"ee38a94e-1190-4f5f-a4f3-2875d2eae79c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:30.590814 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.590490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6kk5\" (UniqueName: \"kubernetes.io/projected/ee38a94e-1190-4f5f-a4f3-2875d2eae79c-kube-api-access-x6kk5\") pod \"monitoring-plugin-6dffbbd67-vv4fj\" (UID: \"ee38a94e-1190-4f5f-a4f3-2875d2eae79c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:30.691086 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.691013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6kk5\" (UniqueName: \"kubernetes.io/projected/ee38a94e-1190-4f5f-a4f3-2875d2eae79c-kube-api-access-x6kk5\") pod \"monitoring-plugin-6dffbbd67-vv4fj\" (UID: \"ee38a94e-1190-4f5f-a4f3-2875d2eae79c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:30.691086 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.691091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ee38a94e-1190-4f5f-a4f3-2875d2eae79c-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vv4fj\" (UID: \"ee38a94e-1190-4f5f-a4f3-2875d2eae79c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:30.693667 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.693635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ee38a94e-1190-4f5f-a4f3-2875d2eae79c-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vv4fj\" (UID: \"ee38a94e-1190-4f5f-a4f3-2875d2eae79c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:30.699190 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.699160 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6kk5\" (UniqueName: \"kubernetes.io/projected/ee38a94e-1190-4f5f-a4f3-2875d2eae79c-kube-api-access-x6kk5\") pod \"monitoring-plugin-6dffbbd67-vv4fj\" (UID: \"ee38a94e-1190-4f5f-a4f3-2875d2eae79c\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:30.812122 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.812018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:30.951922 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:30.951888 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj"] Jul 30 20:22:30.955496 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:30.955468 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee38a94e_1190_4f5f_a4f3_2875d2eae79c.slice/crio-a27a4bc147d56e3cde30ae768e0a274dae061662aa5175deabc6c93900b8b054 WatchSource:0}: Error finding container a27a4bc147d56e3cde30ae768e0a274dae061662aa5175deabc6c93900b8b054: Status 404 returned error can't find the container with id a27a4bc147d56e3cde30ae768e0a274dae061662aa5175deabc6c93900b8b054 Jul 30 20:22:31.367286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.367255 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f5f486fbd-pb6kj"] Jul 30 20:22:31.371550 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.371529 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.375320 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.375296 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 30 20:22:31.375463 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.375376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 30 20:22:31.375606 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.375591 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 30 20:22:31.375667 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.375602 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 30 20:22:31.375667 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.375639 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 30 20:22:31.375667 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.375608 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 30 20:22:31.375848 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.375745 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 30 20:22:31.375912 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.375879 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cm6lb\"" Jul 30 20:22:31.380772 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.380747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5f486fbd-pb6kj"] Jul 30 20:22:31.396353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.396319 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-oauth-config\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.396353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.396357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-oauth-serving-cert\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.396562 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.396380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-serving-cert\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.396562 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.396453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-config\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.396562 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.396493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-service-ca\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.396562 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.396545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhplt\" (UniqueName: \"kubernetes.io/projected/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-kube-api-access-mhplt\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.496986 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.496949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhplt\" (UniqueName: \"kubernetes.io/projected/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-kube-api-access-mhplt\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.496986 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.497003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-oauth-config\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.497284 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.497033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-oauth-serving-cert\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.497284 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.497084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-serving-cert\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.497284 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.497125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-config\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.497284 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.497150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-service-ca\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.498024 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.497974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-service-ca\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.498024 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.497992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-oauth-serving-cert\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.498240 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.498100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-config\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.499937 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.499910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-oauth-config\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.500304 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.500286 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-serving-cert\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.506283 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.506258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhplt\" (UniqueName: \"kubernetes.io/projected/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-kube-api-access-mhplt\") pod \"console-7f5f486fbd-pb6kj\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.551883 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.551840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" event={"ID":"ee38a94e-1190-4f5f-a4f3-2875d2eae79c","Type":"ContainerStarted","Data":"a27a4bc147d56e3cde30ae768e0a274dae061662aa5175deabc6c93900b8b054"} Jul 30 20:22:31.682305 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.682210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:31.816836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:31.816457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5f486fbd-pb6kj"] Jul 30 20:22:31.819755 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:31.819718 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef94815a_d6fc_4aad_b1e0_ed9e861993a4.slice/crio-29d42afa23004a01d8fc998808780c189782e5f061f5977958b9425787fc5c30 WatchSource:0}: Error finding container 29d42afa23004a01d8fc998808780c189782e5f061f5977958b9425787fc5c30: Status 404 returned error can't find the container with id 29d42afa23004a01d8fc998808780c189782e5f061f5977958b9425787fc5c30 Jul 30 20:22:32.557719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:32.557672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5f486fbd-pb6kj" event={"ID":"ef94815a-d6fc-4aad-b1e0-ed9e861993a4","Type":"ContainerStarted","Data":"29d42afa23004a01d8fc998808780c189782e5f061f5977958b9425787fc5c30"} Jul 30 20:22:32.559599 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:32.559536 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" event={"ID":"ee38a94e-1190-4f5f-a4f3-2875d2eae79c","Type":"ContainerStarted","Data":"209e90c250d691c5705f6abe41b136e2b2e3fd3101649afad96776ea9cc6d3ac"} Jul 30 20:22:32.559826 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:32.559796 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:32.565582 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:32.565554 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" Jul 30 20:22:32.580089 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:32.580015 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vv4fj" podStartSLOduration=1.196333311 podStartE2EDuration="2.57999739s" totalImagesPullingTime="1.383664079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:32.579153737 +0000 UTC m=+91.044292084" watchObservedRunningTime="2026-07-30 20:22:32.57999739 +0000 UTC m=+91.045135738" Jul 30 20:22:35.499077 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.499020 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" podUID="b78f9503-3ff3-4800-92ef-da6471ad6071" containerName="registry" containerID="cri-o://21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757" gracePeriod=30 Jul 30 20:22:35.573135 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.573100 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5f486fbd-pb6kj" event={"ID":"ef94815a-d6fc-4aad-b1e0-ed9e861993a4","Type":"ContainerStarted","Data":"72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3"} Jul 30 20:22:35.590441 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.590392 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f5f486fbd-pb6kj" podStartSLOduration=1.67234562 podStartE2EDuration="4.590374824s" totalImagesPullingTime="2.918029204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:35.589445228 +0000 UTC m=+94.054583588" watchObservedRunningTime="2026-07-30 20:22:35.590374824 +0000 UTC m=+94.055513172" Jul 30 20:22:35.735521 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.735495 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:22:35.835934 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.835848 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") pod \"b78f9503-3ff3-4800-92ef-da6471ad6071\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " Jul 30 20:22:35.835934 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.835905 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnf22\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-kube-api-access-jnf22\") pod \"b78f9503-3ff3-4800-92ef-da6471ad6071\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " Jul 30 20:22:35.836197 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.835941 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-trusted-ca\") pod \"b78f9503-3ff3-4800-92ef-da6471ad6071\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " Jul 30 20:22:35.836197 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.835985 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-installation-pull-secrets\") pod \"b78f9503-3ff3-4800-92ef-da6471ad6071\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " Jul 30 20:22:35.836197 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.836023 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b78f9503-3ff3-4800-92ef-da6471ad6071-ca-trust-extracted\") pod \"b78f9503-3ff3-4800-92ef-da6471ad6071\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " Jul 30 20:22:35.836197 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.836049 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-certificates\") pod \"b78f9503-3ff3-4800-92ef-da6471ad6071\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " Jul 30 20:22:35.836197 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.836111 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-image-registry-private-configuration\") pod \"b78f9503-3ff3-4800-92ef-da6471ad6071\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " Jul 30 20:22:35.836197 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.836157 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-bound-sa-token\") pod \"b78f9503-3ff3-4800-92ef-da6471ad6071\" (UID: \"b78f9503-3ff3-4800-92ef-da6471ad6071\") " Jul 30 20:22:35.836559 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.836502 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b78f9503-3ff3-4800-92ef-da6471ad6071" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:22:35.836870 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.836832 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b78f9503-3ff3-4800-92ef-da6471ad6071" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:22:35.838850 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.838811 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b78f9503-3ff3-4800-92ef-da6471ad6071" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:22:35.839169 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.838826 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b78f9503-3ff3-4800-92ef-da6471ad6071" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:22:35.839169 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.838891 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-kube-api-access-jnf22" (OuterVolumeSpecName: "kube-api-access-jnf22") pod "b78f9503-3ff3-4800-92ef-da6471ad6071" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071"). InnerVolumeSpecName "kube-api-access-jnf22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:22:35.839169 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.838900 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b78f9503-3ff3-4800-92ef-da6471ad6071" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:22:35.839169 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.838914 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b78f9503-3ff3-4800-92ef-da6471ad6071" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:22:35.845458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.845427 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78f9503-3ff3-4800-92ef-da6471ad6071-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b78f9503-3ff3-4800-92ef-da6471ad6071" (UID: "b78f9503-3ff3-4800-92ef-da6471ad6071"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:22:35.937410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.937368 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-image-registry-private-configuration\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:22:35.937410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.937402 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-bound-sa-token\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:22:35.937410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.937413 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-tls\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:22:35.937410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.937420 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnf22\" (UniqueName: \"kubernetes.io/projected/b78f9503-3ff3-4800-92ef-da6471ad6071-kube-api-access-jnf22\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:22:35.937675 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.937429 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-trusted-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:22:35.937675 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.937437 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b78f9503-3ff3-4800-92ef-da6471ad6071-installation-pull-secrets\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:22:35.937777 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.937445 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b78f9503-3ff3-4800-92ef-da6471ad6071-ca-trust-extracted\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:22:35.937822 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:35.937783 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b78f9503-3ff3-4800-92ef-da6471ad6071-registry-certificates\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:22:36.577966 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.577929 2575 generic.go:356] "Generic (PLEG): container finished" podID="b78f9503-3ff3-4800-92ef-da6471ad6071" containerID="21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757" exitCode=0 Jul 30 20:22:36.578458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.577989 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" Jul 30 20:22:36.578458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.578024 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" event={"ID":"b78f9503-3ff3-4800-92ef-da6471ad6071","Type":"ContainerDied","Data":"21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757"} Jul 30 20:22:36.578458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.578108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb5fff7f9-2fsh7" event={"ID":"b78f9503-3ff3-4800-92ef-da6471ad6071","Type":"ContainerDied","Data":"bc148c63d1cfad9353a34d5036c47da6c6a0d1154f8d04647413ad98db51b75d"} Jul 30 20:22:36.578458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.578141 2575 scope.go:117] "RemoveContainer" containerID="21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757" Jul 30 20:22:36.586368 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.586346 2575 scope.go:117] "RemoveContainer" containerID="21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757" Jul 30 20:22:36.586677 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:22:36.586652 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757\": container with ID starting with 21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757 not found: ID does not exist" containerID="21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757" Jul 30 20:22:36.586728 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.586694 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757"} err="failed to get container status \"21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757\": rpc error: code = NotFound desc = could not find container \"21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757\": container with ID starting with 21e865681940b9bfe1a9b89289c1915ced648cce7feaf547dca9d663d05ba757 not found: ID does not exist" Jul 30 20:22:36.596149 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.596121 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bb5fff7f9-2fsh7"] Jul 30 20:22:36.599709 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:36.599684 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bb5fff7f9-2fsh7"] Jul 30 20:22:38.012141 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:38.012107 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b78f9503-3ff3-4800-92ef-da6471ad6071" path="/var/lib/kubelet/pods/b78f9503-3ff3-4800-92ef-da6471ad6071/volumes" Jul 30 20:22:40.457008 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:40.456978 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k8266" Jul 30 20:22:41.682376 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:41.682337 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:41.682826 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:41.682441 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:41.687325 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:41.687301 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:42.113785 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.113746 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cb6c5df88-wbrhh"] Jul 30 20:22:42.114082 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.114070 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78f9503-3ff3-4800-92ef-da6471ad6071" containerName="registry" Jul 30 20:22:42.114082 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.114083 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78f9503-3ff3-4800-92ef-da6471ad6071" containerName="registry" Jul 30 20:22:42.114186 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.114162 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78f9503-3ff3-4800-92ef-da6471ad6071" containerName="registry" Jul 30 20:22:42.119630 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.119602 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.127765 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.127737 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 30 20:22:42.127991 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.127878 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb6c5df88-wbrhh"] Jul 30 20:22:42.188995 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.188956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-serving-cert\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.188995 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.188993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-oauth-serving-cert\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.189247 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.189017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-oauth-config\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.189247 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.189145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-trusted-ca-bundle\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.189247 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.189176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-config\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.189247 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.189200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7v54\" (UniqueName: \"kubernetes.io/projected/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-kube-api-access-r7v54\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.189247 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.189243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-service-ca\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.290373 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.290331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-serving-cert\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.290373 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.290372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-oauth-serving-cert\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.290623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.290405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-oauth-config\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.290623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.290459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-trusted-ca-bundle\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.290623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.290493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-config\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.290623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.290522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7v54\" (UniqueName: \"kubernetes.io/projected/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-kube-api-access-r7v54\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.290623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.290561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-service-ca\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.291284 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.291259 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-config\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.291401 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.291349 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-oauth-serving-cert\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.291401 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.291360 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-service-ca\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.291708 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.291685 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-trusted-ca-bundle\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.292888 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.292862 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-oauth-config\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.293230 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.293213 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-serving-cert\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.298914 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.298880 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7v54\" (UniqueName: \"kubernetes.io/projected/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-kube-api-access-r7v54\") pod \"console-cb6c5df88-wbrhh\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.430598 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.430504 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:42.579841 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.579788 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb6c5df88-wbrhh"] Jul 30 20:22:42.582492 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:22:42.582460 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f74e9e9_e6b0_4de1_9b0c_80b1262ba52e.slice/crio-0d9c4aa17037c68f860eafea91fbd53f87c34515bc6e9ee6a6fc4d0855434bed WatchSource:0}: Error finding container 0d9c4aa17037c68f860eafea91fbd53f87c34515bc6e9ee6a6fc4d0855434bed: Status 404 returned error can't find the container with id 0d9c4aa17037c68f860eafea91fbd53f87c34515bc6e9ee6a6fc4d0855434bed Jul 30 20:22:42.603361 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.603321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb6c5df88-wbrhh" event={"ID":"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e","Type":"ContainerStarted","Data":"0d9c4aa17037c68f860eafea91fbd53f87c34515bc6e9ee6a6fc4d0855434bed"} Jul 30 20:22:42.607525 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:42.607498 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:22:43.612700 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:43.612661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb6c5df88-wbrhh" event={"ID":"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e","Type":"ContainerStarted","Data":"f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc"} Jul 30 20:22:43.632559 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:43.632511 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cb6c5df88-wbrhh" podStartSLOduration=1.632494124 podStartE2EDuration="1.632494124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:43.631753276 +0000 UTC m=+102.096891646" watchObservedRunningTime="2026-07-30 20:22:43.632494124 +0000 UTC m=+102.097632472" Jul 30 20:22:52.430800 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:52.430763 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:52.431319 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:52.430842 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:52.435514 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:52.435493 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:52.649148 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:52.649118 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:22:52.698764 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:52.698682 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f5f486fbd-pb6kj"] Jul 30 20:22:57.664425 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:57.664385 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c96510c-70a4-411f-952b-1a4176ab3094" containerID="28a4160e12ab530e9439eb4f6756c91490941026168619c644637e33c140afe8" exitCode=0 Jul 30 20:22:57.664921 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:57.664459 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" event={"ID":"9c96510c-70a4-411f-952b-1a4176ab3094","Type":"ContainerDied","Data":"28a4160e12ab530e9439eb4f6756c91490941026168619c644637e33c140afe8"} Jul 30 20:22:57.664921 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:57.664907 2575 scope.go:117] "RemoveContainer" containerID="28a4160e12ab530e9439eb4f6756c91490941026168619c644637e33c140afe8" Jul 30 20:22:58.670526 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:22:58.670487 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-9c65j" event={"ID":"9c96510c-70a4-411f-952b-1a4176ab3094","Type":"ContainerStarted","Data":"1376cbbe7c39db3f29bfc43132e5ef48728991358f3676f7f0d1a621eb518f47"} Jul 30 20:23:06.701460 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:06.701422 2575 generic.go:356] "Generic (PLEG): container finished" podID="a0f10072-fc2c-4d23-91b3-1a982a9decbd" containerID="a11752fa210a2a853290ffa02bce7c84b4059d10cbba59a2978ecf06d3b4ccb8" exitCode=0 Jul 30 20:23:06.701848 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:06.701495 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" event={"ID":"a0f10072-fc2c-4d23-91b3-1a982a9decbd","Type":"ContainerDied","Data":"a11752fa210a2a853290ffa02bce7c84b4059d10cbba59a2978ecf06d3b4ccb8"} Jul 30 20:23:06.701848 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:06.701803 2575 scope.go:117] "RemoveContainer" containerID="a11752fa210a2a853290ffa02bce7c84b4059d10cbba59a2978ecf06d3b4ccb8" Jul 30 20:23:07.707022 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:07.706988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xx7xc" event={"ID":"a0f10072-fc2c-4d23-91b3-1a982a9decbd","Type":"ContainerStarted","Data":"3b1184b7b60a69d53fd295c4c789281746b6779e51d013cbdbcb85000e14fc92"} Jul 30 20:23:16.739439 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:16.739403 2575 generic.go:356] "Generic (PLEG): container finished" podID="84cde953-2d57-4ce5-8848-8079a5023664" containerID="2f89edd9bcc477a033e52d56153b2b7793b07902f27fdc5841f376e740d8b16b" exitCode=0 Jul 30 20:23:16.739787 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:16.739480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wbkxk" event={"ID":"84cde953-2d57-4ce5-8848-8079a5023664","Type":"ContainerDied","Data":"2f89edd9bcc477a033e52d56153b2b7793b07902f27fdc5841f376e740d8b16b"} Jul 30 20:23:16.739878 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:16.739860 2575 scope.go:117] "RemoveContainer" containerID="2f89edd9bcc477a033e52d56153b2b7793b07902f27fdc5841f376e740d8b16b" Jul 30 20:23:17.723093 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:17.723028 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f5f486fbd-pb6kj" podUID="ef94815a-d6fc-4aad-b1e0-ed9e861993a4" containerName="console" containerID="cri-o://72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3" gracePeriod=15 Jul 30 20:23:17.745287 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:17.745261 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wbkxk" event={"ID":"84cde953-2d57-4ce5-8848-8079a5023664","Type":"ContainerStarted","Data":"098bed849f9c275a74ba9454e13e810b513611a018ff87627ef7410ad67ae5b1"} Jul 30 20:23:17.962258 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:17.962236 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:23:18.082342 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082252 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-service-ca\") pod \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " Jul 30 20:23:18.082342 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082294 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhplt\" (UniqueName: \"kubernetes.io/projected/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-kube-api-access-mhplt\") pod \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " Jul 30 20:23:18.082342 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082337 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-oauth-config\") pod \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " Jul 30 20:23:18.082620 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082452 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-serving-cert\") pod \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " Jul 30 20:23:18.082620 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082500 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-config\") pod \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " Jul 30 20:23:18.082620 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082542 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-oauth-serving-cert\") pod \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\" (UID: \"ef94815a-d6fc-4aad-b1e0-ed9e861993a4\") " Jul 30 20:23:18.082781 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082702 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-service-ca" (OuterVolumeSpecName: "service-ca") pod "ef94815a-d6fc-4aad-b1e0-ed9e861993a4" (UID: "ef94815a-d6fc-4aad-b1e0-ed9e861993a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:23:18.082939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082895 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-config" (OuterVolumeSpecName: "console-config") pod "ef94815a-d6fc-4aad-b1e0-ed9e861993a4" (UID: "ef94815a-d6fc-4aad-b1e0-ed9e861993a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:23:18.082939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082929 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:23:18.083169 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.082951 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ef94815a-d6fc-4aad-b1e0-ed9e861993a4" (UID: "ef94815a-d6fc-4aad-b1e0-ed9e861993a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:23:18.084686 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.084650 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ef94815a-d6fc-4aad-b1e0-ed9e861993a4" (UID: "ef94815a-d6fc-4aad-b1e0-ed9e861993a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:23:18.084775 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.084698 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ef94815a-d6fc-4aad-b1e0-ed9e861993a4" (UID: "ef94815a-d6fc-4aad-b1e0-ed9e861993a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:23:18.084775 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.084720 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-kube-api-access-mhplt" (OuterVolumeSpecName: "kube-api-access-mhplt") pod "ef94815a-d6fc-4aad-b1e0-ed9e861993a4" (UID: "ef94815a-d6fc-4aad-b1e0-ed9e861993a4"). InnerVolumeSpecName "kube-api-access-mhplt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:23:18.183707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.183674 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-serving-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:23:18.183707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.183701 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-config\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:23:18.183707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.183711 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-oauth-serving-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:23:18.183707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.183720 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhplt\" (UniqueName: \"kubernetes.io/projected/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-kube-api-access-mhplt\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:23:18.183994 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.183730 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef94815a-d6fc-4aad-b1e0-ed9e861993a4-console-oauth-config\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:23:18.750435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.750398 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef94815a-d6fc-4aad-b1e0-ed9e861993a4" containerID="72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3" exitCode=0 Jul 30 20:23:18.750847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.750447 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5f486fbd-pb6kj" Jul 30 20:23:18.750847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.750455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5f486fbd-pb6kj" event={"ID":"ef94815a-d6fc-4aad-b1e0-ed9e861993a4","Type":"ContainerDied","Data":"72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3"} Jul 30 20:23:18.750847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.750495 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5f486fbd-pb6kj" event={"ID":"ef94815a-d6fc-4aad-b1e0-ed9e861993a4","Type":"ContainerDied","Data":"29d42afa23004a01d8fc998808780c189782e5f061f5977958b9425787fc5c30"} Jul 30 20:23:18.750847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.750514 2575 scope.go:117] "RemoveContainer" containerID="72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3" Jul 30 20:23:18.759626 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.759605 2575 scope.go:117] "RemoveContainer" containerID="72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3" Jul 30 20:23:18.759861 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:23:18.759833 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3\": container with ID starting with 72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3 not found: ID does not exist" containerID="72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3" Jul 30 20:23:18.759946 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.759868 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3"} err="failed to get container status \"72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3\": rpc error: code = NotFound desc = could not find container \"72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3\": container with ID starting with 72f8920d9c7c537eb8e3c5d75c3ba6cba2310fc8fd074765ff1ac72f66e167e3 not found: ID does not exist" Jul 30 20:23:18.772266 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.772244 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f5f486fbd-pb6kj"] Jul 30 20:23:18.775638 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:18.775619 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f5f486fbd-pb6kj"] Jul 30 20:23:20.011642 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:20.011602 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef94815a-d6fc-4aad-b1e0-ed9e861993a4" path="/var/lib/kubelet/pods/ef94815a-d6fc-4aad-b1e0-ed9e861993a4/volumes" Jul 30 20:23:57.987152 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:57.987041 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-678c664c85-ghk9z"] Jul 30 20:23:57.987666 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:57.987483 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef94815a-d6fc-4aad-b1e0-ed9e861993a4" containerName="console" Jul 30 20:23:57.987666 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:57.987503 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef94815a-d6fc-4aad-b1e0-ed9e861993a4" containerName="console" Jul 30 20:23:57.987666 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:57.987570 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef94815a-d6fc-4aad-b1e0-ed9e861993a4" containerName="console" Jul 30 20:23:57.990561 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:57.990535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:57.999234 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:57.999209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678c664c85-ghk9z"] Jul 30 20:23:58.086166 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.086125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-serving-cert\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.086166 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.086168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-config\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.086412 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.086192 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-oauth-serving-cert\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.086412 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.086258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-oauth-config\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.086412 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.086336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-trusted-ca-bundle\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.086412 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.086390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-service-ca\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.086551 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.086418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj444\" (UniqueName: \"kubernetes.io/projected/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-kube-api-access-xj444\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.187809 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.187765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-service-ca\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.187809 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.187813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj444\" (UniqueName: \"kubernetes.io/projected/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-kube-api-access-xj444\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188088 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.187851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-serving-cert\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188088 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.187868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-config\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188088 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.187987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-oauth-serving-cert\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188088 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.188029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-oauth-config\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188285 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.188098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-trusted-ca-bundle\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188687 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.188659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-service-ca\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188807 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.188670 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-config\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188807 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.188715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-oauth-serving-cert\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.188915 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.188825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-trusted-ca-bundle\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.190603 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.190580 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-serving-cert\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.190722 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.190589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-oauth-config\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.196670 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.196641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj444\" (UniqueName: \"kubernetes.io/projected/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-kube-api-access-xj444\") pod \"console-678c664c85-ghk9z\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.301987 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.301886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:23:58.441547 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.441518 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678c664c85-ghk9z"] Jul 30 20:23:58.444364 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:23:58.444332 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86d9a9f_f7e8_4213_a86e_6cec533f6d3d.slice/crio-ac180ac6ed01499d88d664fa73ec7bd648d5d168e438b54978e46842500d06e0 WatchSource:0}: Error finding container ac180ac6ed01499d88d664fa73ec7bd648d5d168e438b54978e46842500d06e0: Status 404 returned error can't find the container with id ac180ac6ed01499d88d664fa73ec7bd648d5d168e438b54978e46842500d06e0 Jul 30 20:23:58.903589 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.903552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678c664c85-ghk9z" event={"ID":"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d","Type":"ContainerStarted","Data":"359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6"} Jul 30 20:23:58.903770 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.903607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678c664c85-ghk9z" event={"ID":"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d","Type":"ContainerStarted","Data":"ac180ac6ed01499d88d664fa73ec7bd648d5d168e438b54978e46842500d06e0"} Jul 30 20:23:58.920965 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:23:58.920915 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-678c664c85-ghk9z" podStartSLOduration=1.920898125 podStartE2EDuration="1.920898125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:23:58.920409105 +0000 UTC m=+177.385547465" watchObservedRunningTime="2026-07-30 20:23:58.920898125 +0000 UTC m=+177.386036473" Jul 30 20:24:08.302574 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:08.302533 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:24:08.302987 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:08.302622 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:24:08.307476 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:08.307449 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:24:08.945803 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:08.945771 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:24:08.989793 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:08.989753 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb6c5df88-wbrhh"] Jul 30 20:24:34.012268 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.012223 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cb6c5df88-wbrhh" podUID="3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" containerName="console" containerID="cri-o://f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc" gracePeriod=15 Jul 30 20:24:34.242223 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.242199 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:24:34.381612 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.381538 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7v54\" (UniqueName: \"kubernetes.io/projected/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-kube-api-access-r7v54\") pod \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " Jul 30 20:24:34.381612 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.381577 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-oauth-serving-cert\") pod \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " Jul 30 20:24:34.381612 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.381606 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-serving-cert\") pod \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " Jul 30 20:24:34.381827 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.381625 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-config\") pod \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " Jul 30 20:24:34.381827 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.381657 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-oauth-config\") pod \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " Jul 30 20:24:34.381827 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.381734 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-trusted-ca-bundle\") pod \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " Jul 30 20:24:34.381827 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.381761 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-service-ca\") pod \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\" (UID: \"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e\") " Jul 30 20:24:34.382022 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.381967 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" (UID: "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:24:34.382105 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.382078 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-config" (OuterVolumeSpecName: "console-config") pod "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" (UID: "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:24:34.382279 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.382253 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-oauth-serving-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:34.382349 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.382278 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-service-ca" (OuterVolumeSpecName: "service-ca") pod "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" (UID: "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:24:34.382480 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.382462 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" (UID: "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:24:34.383740 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.383722 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-kube-api-access-r7v54" (OuterVolumeSpecName: "kube-api-access-r7v54") pod "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" (UID: "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e"). InnerVolumeSpecName "kube-api-access-r7v54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:24:34.384091 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.384048 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" (UID: "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:24:34.384162 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.384081 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" (UID: "3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:24:34.482715 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.482679 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7v54\" (UniqueName: \"kubernetes.io/projected/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-kube-api-access-r7v54\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:34.482715 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.482711 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-serving-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:34.482715 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.482722 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-config\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:34.482887 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.482731 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-console-oauth-config\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:34.482887 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.482739 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-trusted-ca-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:34.482887 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:34.482747 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:35.037729 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.037694 2575 generic.go:356] "Generic (PLEG): container finished" podID="3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" containerID="f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc" exitCode=0 Jul 30 20:24:35.038190 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.037740 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb6c5df88-wbrhh" Jul 30 20:24:35.038190 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.037786 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb6c5df88-wbrhh" event={"ID":"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e","Type":"ContainerDied","Data":"f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc"} Jul 30 20:24:35.038190 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.037848 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb6c5df88-wbrhh" event={"ID":"3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e","Type":"ContainerDied","Data":"0d9c4aa17037c68f860eafea91fbd53f87c34515bc6e9ee6a6fc4d0855434bed"} Jul 30 20:24:35.038190 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.037878 2575 scope.go:117] "RemoveContainer" containerID="f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc" Jul 30 20:24:35.046202 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.046183 2575 scope.go:117] "RemoveContainer" containerID="f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc" Jul 30 20:24:35.046447 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:24:35.046431 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc\": container with ID starting with f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc not found: ID does not exist" containerID="f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc" Jul 30 20:24:35.046512 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.046453 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc"} err="failed to get container status \"f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc\": rpc error: code = NotFound desc = could not find container \"f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc\": container with ID starting with f0b4ffeddfb7faf432278ffb4ca36c22d07c348d19e1bdce3e622367fe5126dc not found: ID does not exist" Jul 30 20:24:35.058661 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.058636 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb6c5df88-wbrhh"] Jul 30 20:24:35.062791 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:35.062768 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cb6c5df88-wbrhh"] Jul 30 20:24:36.012180 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.012143 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" path="/var/lib/kubelet/pods/3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e/volumes" Jul 30 20:24:36.143854 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.143818 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw"] Jul 30 20:24:36.144221 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.144108 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" containerName="console" Jul 30 20:24:36.144221 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.144119 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" containerName="console" Jul 30 20:24:36.144221 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.144177 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f74e9e9-e6b0-4de1-9b0c-80b1262ba52e" containerName="console" Jul 30 20:24:36.148438 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.148422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.151214 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.151198 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-57lfn\"" Jul 30 20:24:36.151341 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.151320 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 20:24:36.152393 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.152381 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 20:24:36.157145 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.157125 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw"] Jul 30 20:24:36.295115 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.294980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.295115 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.295094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjm4s\" (UniqueName: \"kubernetes.io/projected/4397d686-dbea-481f-b053-c3cf848b5a59-kube-api-access-tjm4s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.295115 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.295121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.396513 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.396465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.396629 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.396530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjm4s\" (UniqueName: \"kubernetes.io/projected/4397d686-dbea-481f-b053-c3cf848b5a59-kube-api-access-tjm4s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.396629 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.396550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.396863 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.396846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.396898 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.396875 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.406230 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.406197 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjm4s\" (UniqueName: \"kubernetes.io/projected/4397d686-dbea-481f-b053-c3cf848b5a59-kube-api-access-tjm4s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.457356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.457302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:36.589730 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:36.589524 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw"] Jul 30 20:24:36.592401 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:24:36.592375 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4397d686_dbea_481f_b053_c3cf848b5a59.slice/crio-ebdb38e0286df336c7433326c06f3fa2f095d558facdf8e680e8b55a939e67c1 WatchSource:0}: Error finding container ebdb38e0286df336c7433326c06f3fa2f095d558facdf8e680e8b55a939e67c1: Status 404 returned error can't find the container with id ebdb38e0286df336c7433326c06f3fa2f095d558facdf8e680e8b55a939e67c1 Jul 30 20:24:37.047719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:37.047687 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" event={"ID":"4397d686-dbea-481f-b053-c3cf848b5a59","Type":"ContainerStarted","Data":"ebdb38e0286df336c7433326c06f3fa2f095d558facdf8e680e8b55a939e67c1"} Jul 30 20:24:43.075458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:43.075406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" event={"ID":"4397d686-dbea-481f-b053-c3cf848b5a59","Type":"ContainerStarted","Data":"479e3f96ebc67f723718e3a02a088ae626b933b9ffd583b60c8d0fc6bdc86069"} Jul 30 20:24:44.080812 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:44.080772 2575 generic.go:356] "Generic (PLEG): container finished" podID="4397d686-dbea-481f-b053-c3cf848b5a59" containerID="479e3f96ebc67f723718e3a02a088ae626b933b9ffd583b60c8d0fc6bdc86069" exitCode=0 Jul 30 20:24:44.081203 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:44.080821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" event={"ID":"4397d686-dbea-481f-b053-c3cf848b5a59","Type":"ContainerDied","Data":"479e3f96ebc67f723718e3a02a088ae626b933b9ffd583b60c8d0fc6bdc86069"} Jul 30 20:24:47.095724 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:47.095689 2575 generic.go:356] "Generic (PLEG): container finished" podID="4397d686-dbea-481f-b053-c3cf848b5a59" containerID="cdb4bd1710e95b93631f05d3b57a10f3cfbf6d6397fae6c1c1a65f25b8cbbdbf" exitCode=0 Jul 30 20:24:47.096118 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:47.095781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" event={"ID":"4397d686-dbea-481f-b053-c3cf848b5a59","Type":"ContainerDied","Data":"cdb4bd1710e95b93631f05d3b57a10f3cfbf6d6397fae6c1c1a65f25b8cbbdbf"} Jul 30 20:24:55.134148 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:55.134045 2575 generic.go:356] "Generic (PLEG): container finished" podID="4397d686-dbea-481f-b053-c3cf848b5a59" containerID="acaa5e1b5c6dc107199e81020592fa5558516e5b7b2d547a39689f03e1cff11b" exitCode=0 Jul 30 20:24:55.134148 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:55.134087 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" event={"ID":"4397d686-dbea-481f-b053-c3cf848b5a59","Type":"ContainerDied","Data":"acaa5e1b5c6dc107199e81020592fa5558516e5b7b2d547a39689f03e1cff11b"} Jul 30 20:24:56.309975 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.309942 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:24:56.346889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.346858 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjm4s\" (UniqueName: \"kubernetes.io/projected/4397d686-dbea-481f-b053-c3cf848b5a59-kube-api-access-tjm4s\") pod \"4397d686-dbea-481f-b053-c3cf848b5a59\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " Jul 30 20:24:56.347043 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.346927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-util\") pod \"4397d686-dbea-481f-b053-c3cf848b5a59\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " Jul 30 20:24:56.347043 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.346971 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-bundle\") pod \"4397d686-dbea-481f-b053-c3cf848b5a59\" (UID: \"4397d686-dbea-481f-b053-c3cf848b5a59\") " Jul 30 20:24:56.347632 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.347604 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-bundle" (OuterVolumeSpecName: "bundle") pod "4397d686-dbea-481f-b053-c3cf848b5a59" (UID: "4397d686-dbea-481f-b053-c3cf848b5a59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:24:56.349209 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.349181 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4397d686-dbea-481f-b053-c3cf848b5a59-kube-api-access-tjm4s" (OuterVolumeSpecName: "kube-api-access-tjm4s") pod "4397d686-dbea-481f-b053-c3cf848b5a59" (UID: "4397d686-dbea-481f-b053-c3cf848b5a59"). InnerVolumeSpecName "kube-api-access-tjm4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:24:56.351195 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.351171 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-util" (OuterVolumeSpecName: "util") pod "4397d686-dbea-481f-b053-c3cf848b5a59" (UID: "4397d686-dbea-481f-b053-c3cf848b5a59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:24:56.448204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.448116 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:56.448204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.448146 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjm4s\" (UniqueName: \"kubernetes.io/projected/4397d686-dbea-481f-b053-c3cf848b5a59-kube-api-access-tjm4s\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:56.448204 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:56.448157 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4397d686-dbea-481f-b053-c3cf848b5a59-util\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:24:57.143433 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:57.143394 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" event={"ID":"4397d686-dbea-481f-b053-c3cf848b5a59","Type":"ContainerDied","Data":"ebdb38e0286df336c7433326c06f3fa2f095d558facdf8e680e8b55a939e67c1"} Jul 30 20:24:57.143624 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:57.143454 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebdb38e0286df336c7433326c06f3fa2f095d558facdf8e680e8b55a939e67c1" Jul 30 20:24:57.143624 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:24:57.143408 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8rnw" Jul 30 20:25:04.044108 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.044046 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7"] Jul 30 20:25:04.044503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.044357 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4397d686-dbea-481f-b053-c3cf848b5a59" containerName="util" Jul 30 20:25:04.044503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.044369 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4397d686-dbea-481f-b053-c3cf848b5a59" containerName="util" Jul 30 20:25:04.044503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.044376 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4397d686-dbea-481f-b053-c3cf848b5a59" containerName="extract" Jul 30 20:25:04.044503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.044381 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4397d686-dbea-481f-b053-c3cf848b5a59" containerName="extract" Jul 30 20:25:04.044503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.044391 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4397d686-dbea-481f-b053-c3cf848b5a59" containerName="pull" Jul 30 20:25:04.044503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.044396 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4397d686-dbea-481f-b053-c3cf848b5a59" containerName="pull" Jul 30 20:25:04.044503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.044443 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4397d686-dbea-481f-b053-c3cf848b5a59" containerName="extract" Jul 30 20:25:04.046788 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.046770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" Jul 30 20:25:04.051203 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.051173 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 30 20:25:04.051311 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.051173 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:25:04.051311 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.051175 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-w98vq\"" Jul 30 20:25:04.060748 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.060711 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7"] Jul 30 20:25:04.103760 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.103717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f29d576-1869-4a1f-86a6-1caa3fce1681-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fmmt7\" (UID: \"3f29d576-1869-4a1f-86a6-1caa3fce1681\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" Jul 30 20:25:04.103760 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.103761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mclg\" (UniqueName: \"kubernetes.io/projected/3f29d576-1869-4a1f-86a6-1caa3fce1681-kube-api-access-7mclg\") pod \"cert-manager-operator-controller-manager-55c444bf86-fmmt7\" (UID: \"3f29d576-1869-4a1f-86a6-1caa3fce1681\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" Jul 30 20:25:04.204544 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.204500 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f29d576-1869-4a1f-86a6-1caa3fce1681-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fmmt7\" (UID: \"3f29d576-1869-4a1f-86a6-1caa3fce1681\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" Jul 30 20:25:04.204544 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.204547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mclg\" (UniqueName: \"kubernetes.io/projected/3f29d576-1869-4a1f-86a6-1caa3fce1681-kube-api-access-7mclg\") pod \"cert-manager-operator-controller-manager-55c444bf86-fmmt7\" (UID: \"3f29d576-1869-4a1f-86a6-1caa3fce1681\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" Jul 30 20:25:04.204890 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.204870 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f29d576-1869-4a1f-86a6-1caa3fce1681-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fmmt7\" (UID: \"3f29d576-1869-4a1f-86a6-1caa3fce1681\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" Jul 30 20:25:04.213265 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.213231 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mclg\" (UniqueName: \"kubernetes.io/projected/3f29d576-1869-4a1f-86a6-1caa3fce1681-kube-api-access-7mclg\") pod \"cert-manager-operator-controller-manager-55c444bf86-fmmt7\" (UID: \"3f29d576-1869-4a1f-86a6-1caa3fce1681\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" Jul 30 20:25:04.355986 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.355900 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" Jul 30 20:25:04.507731 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:04.507691 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7"] Jul 30 20:25:04.511126 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:25:04.511088 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f29d576_1869_4a1f_86a6_1caa3fce1681.slice/crio-60c19d92650c162098451a1a08c5956ce359e12647a1e2ccda2f4406da103ec1 WatchSource:0}: Error finding container 60c19d92650c162098451a1a08c5956ce359e12647a1e2ccda2f4406da103ec1: Status 404 returned error can't find the container with id 60c19d92650c162098451a1a08c5956ce359e12647a1e2ccda2f4406da103ec1 Jul 30 20:25:05.172813 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:05.172774 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" event={"ID":"3f29d576-1869-4a1f-86a6-1caa3fce1681","Type":"ContainerStarted","Data":"60c19d92650c162098451a1a08c5956ce359e12647a1e2ccda2f4406da103ec1"} Jul 30 20:25:07.183826 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.183730 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" event={"ID":"3f29d576-1869-4a1f-86a6-1caa3fce1681","Type":"ContainerStarted","Data":"b86a7dd45a7e0d159fdf38cf819031fc5c432d7bcf0117fe9aa0e521ebefd023"} Jul 30 20:25:07.210330 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.210270 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fmmt7" podStartSLOduration=0.967314801 podStartE2EDuration="3.210248809s" totalImagesPullingTime="2.242934008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:07.208905444 +0000 UTC m=+245.674043789" watchObservedRunningTime="2026-07-30 20:25:07.210248809 +0000 UTC m=+245.675387155" Jul 30 20:25:07.829327 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.829285 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2tmdf"] Jul 30 20:25:07.831520 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.831494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:07.834843 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.834815 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 30 20:25:07.834978 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.834844 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 30 20:25:07.834978 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.834915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mdqrm\"" Jul 30 20:25:07.850355 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.850317 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2tmdf"] Jul 30 20:25:07.935397 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.935356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64ea0574-51c7-44ff-aad8-cd5a2bae849e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2tmdf\" (UID: \"64ea0574-51c7-44ff-aad8-cd5a2bae849e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:07.935397 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:07.935399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6wq\" (UniqueName: \"kubernetes.io/projected/64ea0574-51c7-44ff-aad8-cd5a2bae849e-kube-api-access-rk6wq\") pod \"cert-manager-webhook-6486ffd997-2tmdf\" (UID: \"64ea0574-51c7-44ff-aad8-cd5a2bae849e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:08.036342 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.036294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64ea0574-51c7-44ff-aad8-cd5a2bae849e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2tmdf\" (UID: \"64ea0574-51c7-44ff-aad8-cd5a2bae849e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:08.036342 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.036345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6wq\" (UniqueName: \"kubernetes.io/projected/64ea0574-51c7-44ff-aad8-cd5a2bae849e-kube-api-access-rk6wq\") pod \"cert-manager-webhook-6486ffd997-2tmdf\" (UID: \"64ea0574-51c7-44ff-aad8-cd5a2bae849e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:08.047941 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.047909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64ea0574-51c7-44ff-aad8-cd5a2bae849e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2tmdf\" (UID: \"64ea0574-51c7-44ff-aad8-cd5a2bae849e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:08.048099 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.047975 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6wq\" (UniqueName: \"kubernetes.io/projected/64ea0574-51c7-44ff-aad8-cd5a2bae849e-kube-api-access-rk6wq\") pod \"cert-manager-webhook-6486ffd997-2tmdf\" (UID: \"64ea0574-51c7-44ff-aad8-cd5a2bae849e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:08.156298 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.156183 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:08.288509 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.288480 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2tmdf"] Jul 30 20:25:08.290833 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:25:08.290807 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64ea0574_51c7_44ff_aad8_cd5a2bae849e.slice/crio-52f1307d561f2ac3325953e6c5e106edcba6428a6f79bcfc149840dca294bdc5 WatchSource:0}: Error finding container 52f1307d561f2ac3325953e6c5e106edcba6428a6f79bcfc149840dca294bdc5: Status 404 returned error can't find the container with id 52f1307d561f2ac3325953e6c5e106edcba6428a6f79bcfc149840dca294bdc5 Jul 30 20:25:08.476399 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.476367 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg"] Jul 30 20:25:08.479463 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.479446 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.482194 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.482167 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-57lfn\"" Jul 30 20:25:08.482288 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.482206 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 20:25:08.482335 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.482290 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 20:25:08.489817 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.489792 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg"] Jul 30 20:25:08.539509 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.539467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclrj\" (UniqueName: \"kubernetes.io/projected/5465066a-de1b-4162-b618-11a09db00906-kube-api-access-xclrj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.539659 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.539560 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.539659 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.539590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.640815 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.640779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.640815 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.640818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.641104 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.640867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclrj\" (UniqueName: \"kubernetes.io/projected/5465066a-de1b-4162-b618-11a09db00906-kube-api-access-xclrj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.641308 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.641286 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.641385 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.641288 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.649097 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.649076 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclrj\" (UniqueName: \"kubernetes.io/projected/5465066a-de1b-4162-b618-11a09db00906-kube-api-access-xclrj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.789383 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.789284 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:08.922139 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:08.922117 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg"] Jul 30 20:25:08.924503 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:25:08.924468 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5465066a_de1b_4162_b618_11a09db00906.slice/crio-8574a3eb2df0817a34f87fd10fae0dc025dea90058733a5f531d5f7f8536fcc8 WatchSource:0}: Error finding container 8574a3eb2df0817a34f87fd10fae0dc025dea90058733a5f531d5f7f8536fcc8: Status 404 returned error can't find the container with id 8574a3eb2df0817a34f87fd10fae0dc025dea90058733a5f531d5f7f8536fcc8 Jul 30 20:25:09.194440 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:09.194343 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" event={"ID":"64ea0574-51c7-44ff-aad8-cd5a2bae849e","Type":"ContainerStarted","Data":"52f1307d561f2ac3325953e6c5e106edcba6428a6f79bcfc149840dca294bdc5"} Jul 30 20:25:09.195687 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:09.195659 2575 generic.go:356] "Generic (PLEG): container finished" podID="5465066a-de1b-4162-b618-11a09db00906" containerID="ad2872d12c3f1edc5230e296970616b127387eaf4473a120899992da60cf889c" exitCode=0 Jul 30 20:25:09.195687 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:09.195689 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" event={"ID":"5465066a-de1b-4162-b618-11a09db00906","Type":"ContainerDied","Data":"ad2872d12c3f1edc5230e296970616b127387eaf4473a120899992da60cf889c"} Jul 30 20:25:09.195825 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:09.195716 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" event={"ID":"5465066a-de1b-4162-b618-11a09db00906","Type":"ContainerStarted","Data":"8574a3eb2df0817a34f87fd10fae0dc025dea90058733a5f531d5f7f8536fcc8"} Jul 30 20:25:12.211503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:12.211470 2575 generic.go:356] "Generic (PLEG): container finished" podID="5465066a-de1b-4162-b618-11a09db00906" containerID="e2f95bd2b2bc168ce76f737d0f099b9c6dffe8bcea825f50477f24a3bad6698b" exitCode=0 Jul 30 20:25:12.211985 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:12.211552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" event={"ID":"5465066a-de1b-4162-b618-11a09db00906","Type":"ContainerDied","Data":"e2f95bd2b2bc168ce76f737d0f099b9c6dffe8bcea825f50477f24a3bad6698b"} Jul 30 20:25:12.213239 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:12.213213 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" event={"ID":"64ea0574-51c7-44ff-aad8-cd5a2bae849e","Type":"ContainerStarted","Data":"9ca841a08f03352c7898463ee6830c43c724ed2a5ed85ee900aa57b9dfea25c8"} Jul 30 20:25:12.213356 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:12.213339 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:12.243100 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:12.243033 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" podStartSLOduration=1.61897789 podStartE2EDuration="5.243016634s" totalImagesPullingTime="3.624038744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:12.242670301 +0000 UTC m=+250.707808647" watchObservedRunningTime="2026-07-30 20:25:12.243016634 +0000 UTC m=+250.708154980" Jul 30 20:25:13.220779 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:13.220746 2575 generic.go:356] "Generic (PLEG): container finished" podID="5465066a-de1b-4162-b618-11a09db00906" containerID="4f8bca2e46c1932f9008ef9833ec91af4be224ef0aa6add94535a9e882518144" exitCode=0 Jul 30 20:25:13.221160 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:13.220832 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" event={"ID":"5465066a-de1b-4162-b618-11a09db00906","Type":"ContainerDied","Data":"4f8bca2e46c1932f9008ef9833ec91af4be224ef0aa6add94535a9e882518144"} Jul 30 20:25:14.352703 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.352677 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:14.385135 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.385109 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xclrj\" (UniqueName: \"kubernetes.io/projected/5465066a-de1b-4162-b618-11a09db00906-kube-api-access-xclrj\") pod \"5465066a-de1b-4162-b618-11a09db00906\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " Jul 30 20:25:14.385288 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.385155 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-bundle\") pod \"5465066a-de1b-4162-b618-11a09db00906\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " Jul 30 20:25:14.385288 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.385227 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-util\") pod \"5465066a-de1b-4162-b618-11a09db00906\" (UID: \"5465066a-de1b-4162-b618-11a09db00906\") " Jul 30 20:25:14.385599 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.385576 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-bundle" (OuterVolumeSpecName: "bundle") pod "5465066a-de1b-4162-b618-11a09db00906" (UID: "5465066a-de1b-4162-b618-11a09db00906"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:25:14.387325 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.387296 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5465066a-de1b-4162-b618-11a09db00906-kube-api-access-xclrj" (OuterVolumeSpecName: "kube-api-access-xclrj") pod "5465066a-de1b-4162-b618-11a09db00906" (UID: "5465066a-de1b-4162-b618-11a09db00906"). InnerVolumeSpecName "kube-api-access-xclrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:25:14.392194 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.392165 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-util" (OuterVolumeSpecName: "util") pod "5465066a-de1b-4162-b618-11a09db00906" (UID: "5465066a-de1b-4162-b618-11a09db00906"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:25:14.486669 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.486576 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:25:14.486669 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.486610 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5465066a-de1b-4162-b618-11a09db00906-util\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:25:14.486669 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:14.486624 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xclrj\" (UniqueName: \"kubernetes.io/projected/5465066a-de1b-4162-b618-11a09db00906-kube-api-access-xclrj\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:25:15.230664 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:15.230636 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" Jul 30 20:25:15.230664 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:15.230648 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgwddg" event={"ID":"5465066a-de1b-4162-b618-11a09db00906","Type":"ContainerDied","Data":"8574a3eb2df0817a34f87fd10fae0dc025dea90058733a5f531d5f7f8536fcc8"} Jul 30 20:25:15.230861 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:15.230690 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8574a3eb2df0817a34f87fd10fae0dc025dea90058733a5f531d5f7f8536fcc8" Jul 30 20:25:17.756076 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.756021 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r2jqv"] Jul 30 20:25:17.756446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.756337 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5465066a-de1b-4162-b618-11a09db00906" containerName="pull" Jul 30 20:25:17.756446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.756349 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5465066a-de1b-4162-b618-11a09db00906" containerName="pull" Jul 30 20:25:17.756446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.756362 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5465066a-de1b-4162-b618-11a09db00906" containerName="util" Jul 30 20:25:17.756446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.756368 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5465066a-de1b-4162-b618-11a09db00906" containerName="util" Jul 30 20:25:17.756446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.756374 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5465066a-de1b-4162-b618-11a09db00906" containerName="extract" Jul 30 20:25:17.756446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.756379 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5465066a-de1b-4162-b618-11a09db00906" containerName="extract" Jul 30 20:25:17.756446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.756432 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5465066a-de1b-4162-b618-11a09db00906" containerName="extract" Jul 30 20:25:17.760425 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.760408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-r2jqv" Jul 30 20:25:17.763137 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.763108 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-m7dz7\"" Jul 30 20:25:17.767753 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.767730 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r2jqv"] Jul 30 20:25:17.813370 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.813336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168c3edd-7cb0-4c12-ba28-86d720490fe1-bound-sa-token\") pod \"cert-manager-55b4d954d5-r2jqv\" (UID: \"168c3edd-7cb0-4c12-ba28-86d720490fe1\") " pod="cert-manager/cert-manager-55b4d954d5-r2jqv" Jul 30 20:25:17.813527 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.813392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcl7\" (UniqueName: \"kubernetes.io/projected/168c3edd-7cb0-4c12-ba28-86d720490fe1-kube-api-access-zhcl7\") pod \"cert-manager-55b4d954d5-r2jqv\" (UID: \"168c3edd-7cb0-4c12-ba28-86d720490fe1\") " pod="cert-manager/cert-manager-55b4d954d5-r2jqv" Jul 30 20:25:17.913860 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.913825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168c3edd-7cb0-4c12-ba28-86d720490fe1-bound-sa-token\") pod \"cert-manager-55b4d954d5-r2jqv\" (UID: \"168c3edd-7cb0-4c12-ba28-86d720490fe1\") " pod="cert-manager/cert-manager-55b4d954d5-r2jqv" Jul 30 20:25:17.914041 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.913905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcl7\" (UniqueName: \"kubernetes.io/projected/168c3edd-7cb0-4c12-ba28-86d720490fe1-kube-api-access-zhcl7\") pod \"cert-manager-55b4d954d5-r2jqv\" (UID: \"168c3edd-7cb0-4c12-ba28-86d720490fe1\") " pod="cert-manager/cert-manager-55b4d954d5-r2jqv" Jul 30 20:25:17.925407 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.925373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168c3edd-7cb0-4c12-ba28-86d720490fe1-bound-sa-token\") pod \"cert-manager-55b4d954d5-r2jqv\" (UID: \"168c3edd-7cb0-4c12-ba28-86d720490fe1\") " pod="cert-manager/cert-manager-55b4d954d5-r2jqv" Jul 30 20:25:17.925536 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:17.925486 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcl7\" (UniqueName: \"kubernetes.io/projected/168c3edd-7cb0-4c12-ba28-86d720490fe1-kube-api-access-zhcl7\") pod \"cert-manager-55b4d954d5-r2jqv\" (UID: \"168c3edd-7cb0-4c12-ba28-86d720490fe1\") " pod="cert-manager/cert-manager-55b4d954d5-r2jqv" Jul 30 20:25:18.070034 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:18.069934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-r2jqv" Jul 30 20:25:18.200152 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:18.200117 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r2jqv"] Jul 30 20:25:18.203358 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:25:18.203327 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod168c3edd_7cb0_4c12_ba28_86d720490fe1.slice/crio-2ea56f3771118afd4330a9823c1924d9f352c70a5ab5959e61d1e52be4ea4c1f WatchSource:0}: Error finding container 2ea56f3771118afd4330a9823c1924d9f352c70a5ab5959e61d1e52be4ea4c1f: Status 404 returned error can't find the container with id 2ea56f3771118afd4330a9823c1924d9f352c70a5ab5959e61d1e52be4ea4c1f Jul 30 20:25:18.222700 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:18.222679 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2tmdf" Jul 30 20:25:18.244071 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:18.244029 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-r2jqv" event={"ID":"168c3edd-7cb0-4c12-ba28-86d720490fe1","Type":"ContainerStarted","Data":"2ea56f3771118afd4330a9823c1924d9f352c70a5ab5959e61d1e52be4ea4c1f"} Jul 30 20:25:19.249409 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:19.249374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-r2jqv" event={"ID":"168c3edd-7cb0-4c12-ba28-86d720490fe1","Type":"ContainerStarted","Data":"cf6bc4f64c79b2109fa4fe516a7dc241776d1c7a7693f05997531ca9f53ec3bc"} Jul 30 20:25:19.266465 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:19.266411 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-r2jqv" podStartSLOduration=2.2663949199999998 podStartE2EDuration="2.26639492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:19.265049708 +0000 UTC m=+257.730188056" watchObservedRunningTime="2026-07-30 20:25:19.26639492 +0000 UTC m=+257.731533267" Jul 30 20:25:28.440261 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.440177 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf"] Jul 30 20:25:28.491354 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.491318 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf"] Jul 30 20:25:28.491524 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.491446 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.494238 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.494215 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-57lfn\"" Jul 30 20:25:28.494368 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.494240 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 20:25:28.495177 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.495156 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 20:25:28.606393 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.606359 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6pt\" (UniqueName: \"kubernetes.io/projected/781f9917-d949-4105-9d04-d40ec49caac3-kube-api-access-9t6pt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.606573 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.606412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.606573 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.606457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.707723 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.707688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6pt\" (UniqueName: \"kubernetes.io/projected/781f9917-d949-4105-9d04-d40ec49caac3-kube-api-access-9t6pt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.707893 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.707750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.707893 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.707785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.708162 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.708142 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.708248 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.708178 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.715417 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.715389 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6pt\" (UniqueName: \"kubernetes.io/projected/781f9917-d949-4105-9d04-d40ec49caac3-kube-api-access-9t6pt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.801037 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.800998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:28.951199 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:28.950946 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf"] Jul 30 20:25:28.953783 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:25:28.953748 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781f9917_d949_4105_9d04_d40ec49caac3.slice/crio-828c3d8d5713f0ad1b98ae116206b11d5c329e4541769bbdf8a8a490e0391a8c WatchSource:0}: Error finding container 828c3d8d5713f0ad1b98ae116206b11d5c329e4541769bbdf8a8a490e0391a8c: Status 404 returned error can't find the container with id 828c3d8d5713f0ad1b98ae116206b11d5c329e4541769bbdf8a8a490e0391a8c Jul 30 20:25:29.295385 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:29.295294 2575 generic.go:356] "Generic (PLEG): container finished" podID="781f9917-d949-4105-9d04-d40ec49caac3" containerID="5979f3a9fe1e81529e77582d3e199152fb54c58a86f929da37136211c45b5297" exitCode=0 Jul 30 20:25:29.295535 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:29.295376 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" event={"ID":"781f9917-d949-4105-9d04-d40ec49caac3","Type":"ContainerDied","Data":"5979f3a9fe1e81529e77582d3e199152fb54c58a86f929da37136211c45b5297"} Jul 30 20:25:29.295535 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:29.295431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" event={"ID":"781f9917-d949-4105-9d04-d40ec49caac3","Type":"ContainerStarted","Data":"828c3d8d5713f0ad1b98ae116206b11d5c329e4541769bbdf8a8a490e0391a8c"} Jul 30 20:25:31.306311 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:31.306275 2575 generic.go:356] "Generic (PLEG): container finished" podID="781f9917-d949-4105-9d04-d40ec49caac3" containerID="961810ad98081e13a7cc92112eecb4a4e1a126835705a3c35822b64814e21a8d" exitCode=0 Jul 30 20:25:31.306712 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:31.306362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" event={"ID":"781f9917-d949-4105-9d04-d40ec49caac3","Type":"ContainerDied","Data":"961810ad98081e13a7cc92112eecb4a4e1a126835705a3c35822b64814e21a8d"} Jul 30 20:25:32.313036 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:32.313002 2575 generic.go:356] "Generic (PLEG): container finished" podID="781f9917-d949-4105-9d04-d40ec49caac3" containerID="2b311c2a16f1a317ce504d2921597d498aa2e4cddc09874bd20d397e62a8d882" exitCode=0 Jul 30 20:25:32.313457 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:32.313090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" event={"ID":"781f9917-d949-4105-9d04-d40ec49caac3","Type":"ContainerDied","Data":"2b311c2a16f1a317ce504d2921597d498aa2e4cddc09874bd20d397e62a8d882"} Jul 30 20:25:33.442274 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.442251 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:33.550253 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.550219 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-util\") pod \"781f9917-d949-4105-9d04-d40ec49caac3\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " Jul 30 20:25:33.550433 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.550311 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-bundle\") pod \"781f9917-d949-4105-9d04-d40ec49caac3\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " Jul 30 20:25:33.550433 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.550338 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t6pt\" (UniqueName: \"kubernetes.io/projected/781f9917-d949-4105-9d04-d40ec49caac3-kube-api-access-9t6pt\") pod \"781f9917-d949-4105-9d04-d40ec49caac3\" (UID: \"781f9917-d949-4105-9d04-d40ec49caac3\") " Jul 30 20:25:33.551003 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.550973 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-bundle" (OuterVolumeSpecName: "bundle") pod "781f9917-d949-4105-9d04-d40ec49caac3" (UID: "781f9917-d949-4105-9d04-d40ec49caac3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:25:33.552415 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.552393 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781f9917-d949-4105-9d04-d40ec49caac3-kube-api-access-9t6pt" (OuterVolumeSpecName: "kube-api-access-9t6pt") pod "781f9917-d949-4105-9d04-d40ec49caac3" (UID: "781f9917-d949-4105-9d04-d40ec49caac3"). InnerVolumeSpecName "kube-api-access-9t6pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:25:33.555962 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.555937 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-util" (OuterVolumeSpecName: "util") pod "781f9917-d949-4105-9d04-d40ec49caac3" (UID: "781f9917-d949-4105-9d04-d40ec49caac3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:25:33.651652 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.651558 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-util\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:25:33.651652 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.651600 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781f9917-d949-4105-9d04-d40ec49caac3-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:25:33.651652 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:33.651609 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6pt\" (UniqueName: \"kubernetes.io/projected/781f9917-d949-4105-9d04-d40ec49caac3-kube-api-access-9t6pt\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:25:34.323621 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:34.323527 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" event={"ID":"781f9917-d949-4105-9d04-d40ec49caac3","Type":"ContainerDied","Data":"828c3d8d5713f0ad1b98ae116206b11d5c329e4541769bbdf8a8a490e0391a8c"} Jul 30 20:25:34.323621 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:34.323550 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhnlf" Jul 30 20:25:34.323621 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:34.323579 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="828c3d8d5713f0ad1b98ae116206b11d5c329e4541769bbdf8a8a490e0391a8c" Jul 30 20:25:37.814268 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.814235 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr"] Jul 30 20:25:37.814656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.814513 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781f9917-d949-4105-9d04-d40ec49caac3" containerName="util" Jul 30 20:25:37.814656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.814523 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="781f9917-d949-4105-9d04-d40ec49caac3" containerName="util" Jul 30 20:25:37.814656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.814537 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781f9917-d949-4105-9d04-d40ec49caac3" containerName="extract" Jul 30 20:25:37.814656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.814542 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="781f9917-d949-4105-9d04-d40ec49caac3" containerName="extract" Jul 30 20:25:37.814656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.814559 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781f9917-d949-4105-9d04-d40ec49caac3" containerName="pull" Jul 30 20:25:37.814656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.814564 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="781f9917-d949-4105-9d04-d40ec49caac3" containerName="pull" Jul 30 20:25:37.814656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.814608 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="781f9917-d949-4105-9d04-d40ec49caac3" containerName="extract" Jul 30 20:25:37.816493 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.816474 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.819262 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.819232 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 30 20:25:37.820505 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.820481 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 30 20:25:37.820810 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.820786 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8hnzv\"" Jul 30 20:25:37.820929 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.820906 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 30 20:25:37.821069 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.821036 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:25:37.821399 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.821379 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 30 20:25:37.825994 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.825969 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr"] Jul 30 20:25:37.888148 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.888104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/915c3e37-ec0b-4587-b834-ed7e657eebf0-cert\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.888326 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.888157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncq8\" (UniqueName: \"kubernetes.io/projected/915c3e37-ec0b-4587-b834-ed7e657eebf0-kube-api-access-6ncq8\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.888326 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.888254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/915c3e37-ec0b-4587-b834-ed7e657eebf0-manager-config\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.888326 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.888280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/915c3e37-ec0b-4587-b834-ed7e657eebf0-metrics-cert\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.989485 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.989443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/915c3e37-ec0b-4587-b834-ed7e657eebf0-manager-config\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.989485 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.989485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/915c3e37-ec0b-4587-b834-ed7e657eebf0-metrics-cert\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.989685 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.989618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/915c3e37-ec0b-4587-b834-ed7e657eebf0-cert\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.989725 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.989681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncq8\" (UniqueName: \"kubernetes.io/projected/915c3e37-ec0b-4587-b834-ed7e657eebf0-kube-api-access-6ncq8\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.990312 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.990287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/915c3e37-ec0b-4587-b834-ed7e657eebf0-manager-config\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.991977 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.991949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/915c3e37-ec0b-4587-b834-ed7e657eebf0-cert\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.992111 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.992084 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/915c3e37-ec0b-4587-b834-ed7e657eebf0-metrics-cert\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:37.998384 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:37.998354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncq8\" (UniqueName: \"kubernetes.io/projected/915c3e37-ec0b-4587-b834-ed7e657eebf0-kube-api-access-6ncq8\") pod \"lws-controller-manager-696d745bf6-rntwr\" (UID: \"915c3e37-ec0b-4587-b834-ed7e657eebf0\") " pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:38.130544 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:38.130450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:38.264515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:38.264490 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr"] Jul 30 20:25:38.267279 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:25:38.267248 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915c3e37_ec0b_4587_b834_ed7e657eebf0.slice/crio-58b92a193fe23727834117d5fe387545eaa6fca1913ad103888891c05f888e2f WatchSource:0}: Error finding container 58b92a193fe23727834117d5fe387545eaa6fca1913ad103888891c05f888e2f: Status 404 returned error can't find the container with id 58b92a193fe23727834117d5fe387545eaa6fca1913ad103888891c05f888e2f Jul 30 20:25:38.340759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:38.340718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" event={"ID":"915c3e37-ec0b-4587-b834-ed7e657eebf0","Type":"ContainerStarted","Data":"58b92a193fe23727834117d5fe387545eaa6fca1913ad103888891c05f888e2f"} Jul 30 20:25:40.352112 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:40.352076 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" event={"ID":"915c3e37-ec0b-4587-b834-ed7e657eebf0","Type":"ContainerStarted","Data":"83325559f4fed94d9f554b0ce26cf0a92b64dcc0450b5903323153806a56c357"} Jul 30 20:25:40.352457 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:40.352137 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:40.370097 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:40.370030 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" podStartSLOduration=1.919243872 podStartE2EDuration="3.370013315s" totalImagesPullingTime="1.450769443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:40.368976764 +0000 UTC m=+278.834115113" watchObservedRunningTime="2026-07-30 20:25:40.370013315 +0000 UTC m=+278.835151663" Jul 30 20:25:41.995709 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:41.995676 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq"] Jul 30 20:25:41.997824 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:41.997804 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.001856 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.001832 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 20:25:42.002028 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.001911 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 20:25:42.002028 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.002018 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 20:25:42.002207 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.002167 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wcr5p\"" Jul 30 20:25:42.002268 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.002213 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 20:25:42.014458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.014427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq"] Jul 30 20:25:42.122940 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.122899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21995964-ee21-4990-886e-9e6c06b5bd0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.123152 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.122956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21995964-ee21-4990-886e-9e6c06b5bd0d-manifests\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.123152 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.122978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7pz6\" (UniqueName: \"kubernetes.io/projected/21995964-ee21-4990-886e-9e6c06b5bd0d-kube-api-access-b7pz6\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.123152 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.123003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21995964-ee21-4990-886e-9e6c06b5bd0d-tmp\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.123152 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.123031 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21995964-ee21-4990-886e-9e6c06b5bd0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.226858 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.224749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21995964-ee21-4990-886e-9e6c06b5bd0d-tmp\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.226858 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.224846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21995964-ee21-4990-886e-9e6c06b5bd0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.226858 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.224881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21995964-ee21-4990-886e-9e6c06b5bd0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.226858 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.224962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21995964-ee21-4990-886e-9e6c06b5bd0d-manifests\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.226858 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.225002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7pz6\" (UniqueName: \"kubernetes.io/projected/21995964-ee21-4990-886e-9e6c06b5bd0d-kube-api-access-b7pz6\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.230270 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.228369 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21995964-ee21-4990-886e-9e6c06b5bd0d-tmp\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.230270 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.230208 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21995964-ee21-4990-886e-9e6c06b5bd0d-manifests\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.231508 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.231468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21995964-ee21-4990-886e-9e6c06b5bd0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.232260 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.232238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21995964-ee21-4990-886e-9e6c06b5bd0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.233125 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.233106 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7pz6\" (UniqueName: \"kubernetes.io/projected/21995964-ee21-4990-886e-9e6c06b5bd0d-kube-api-access-b7pz6\") pod \"opendatahub-operator-controller-manager-58fb4f68b-7grfq\" (UID: \"21995964-ee21-4990-886e-9e6c06b5bd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.308362 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.308259 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:42.473399 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:42.473363 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq"] Jul 30 20:25:42.477124 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:25:42.477049 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21995964_ee21_4990_886e_9e6c06b5bd0d.slice/crio-ba1df92eb1e767e7ad33157e7125f3d760bb185a27b25a268766338c31e02bf0 WatchSource:0}: Error finding container ba1df92eb1e767e7ad33157e7125f3d760bb185a27b25a268766338c31e02bf0: Status 404 returned error can't find the container with id ba1df92eb1e767e7ad33157e7125f3d760bb185a27b25a268766338c31e02bf0 Jul 30 20:25:43.369980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:43.369942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" event={"ID":"21995964-ee21-4990-886e-9e6c06b5bd0d","Type":"ContainerStarted","Data":"ba1df92eb1e767e7ad33157e7125f3d760bb185a27b25a268766338c31e02bf0"} Jul 30 20:25:46.384811 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:46.384776 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" event={"ID":"21995964-ee21-4990-886e-9e6c06b5bd0d","Type":"ContainerStarted","Data":"9c146d0289ad6614e069090a36aee10676064b167155b2f14a2a9661f6ebc847"} Jul 30 20:25:47.391218 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:47.391181 2575 generic.go:356] "Generic (PLEG): container finished" podID="21995964-ee21-4990-886e-9e6c06b5bd0d" containerID="9c146d0289ad6614e069090a36aee10676064b167155b2f14a2a9661f6ebc847" exitCode=0 Jul 30 20:25:47.391600 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:47.391269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" event={"ID":"21995964-ee21-4990-886e-9e6c06b5bd0d","Type":"ContainerDied","Data":"9c146d0289ad6614e069090a36aee10676064b167155b2f14a2a9661f6ebc847"} Jul 30 20:25:48.398425 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:48.398379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" event={"ID":"21995964-ee21-4990-886e-9e6c06b5bd0d","Type":"ContainerStarted","Data":"2a1479dcb9b625ba818857215014ba4b1b37da72d07d8802b72bf367ddafef70"} Jul 30 20:25:48.398812 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:48.398465 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:25:48.421754 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:48.421705 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" podStartSLOduration=3.054256199 podStartE2EDuration="7.42168658s" totalImagesPullingTime="3.35760885s" totalInitContainerRuntime="1.009821531s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:48.418974839 +0000 UTC m=+286.884113198" watchObservedRunningTime="2026-07-30 20:25:48.42168658 +0000 UTC m=+286.886824928" Jul 30 20:25:51.359003 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:51.358972 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-696d745bf6-rntwr" Jul 30 20:25:59.406028 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:25:59.405995 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-7grfq" Jul 30 20:26:01.960933 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:01.960896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:26:01.966743 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:01.966710 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:26:01.975602 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:01.975575 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 20:26:09.018823 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.018785 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6"] Jul 30 20:26:09.020852 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.020835 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.023466 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.023444 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 30 20:26:09.023652 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.023640 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 20:26:09.023825 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.023809 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 30 20:26:09.024122 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.024102 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 30 20:26:09.024206 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.024188 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-89gr7\"" Jul 30 20:26:09.030822 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.030803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98bb757-2b33-4ae5-944e-abc2a991b0df-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.030901 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.030842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.030901 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.030861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.030901 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.030881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.030997 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.030903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.030997 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.030925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vk9r\" (UniqueName: \"kubernetes.io/projected/c98bb757-2b33-4ae5-944e-abc2a991b0df-kube-api-access-9vk9r\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.030997 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.030949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.034042 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.034016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6"] Jul 30 20:26:09.131803 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.131764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.131992 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.131813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vk9r\" (UniqueName: \"kubernetes.io/projected/c98bb757-2b33-4ae5-944e-abc2a991b0df-kube-api-access-9vk9r\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.131992 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.131857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.131992 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.131886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98bb757-2b33-4ae5-944e-abc2a991b0df-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.131992 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.131940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.131992 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.131961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.132291 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.131992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.132587 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.132548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.134466 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.134442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98bb757-2b33-4ae5-944e-abc2a991b0df-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.134571 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.134487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.134617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.134571 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.134656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.134643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.140339 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.140299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98bb757-2b33-4ae5-944e-abc2a991b0df-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.140339 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.140326 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vk9r\" (UniqueName: \"kubernetes.io/projected/c98bb757-2b33-4ae5-944e-abc2a991b0df-kube-api-access-9vk9r\") pod \"istiod-openshift-gateway-5677ccb49b-54ds6\" (UID: \"c98bb757-2b33-4ae5-944e-abc2a991b0df\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.330448 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.330342 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:09.473396 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.473358 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6"] Jul 30 20:26:09.476548 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:26:09.476518 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98bb757_2b33_4ae5_944e_abc2a991b0df.slice/crio-651c4057a973ae689ee4f84657504d71f867f91fbe7684eb48b4ab4aaf3612ae WatchSource:0}: Error finding container 651c4057a973ae689ee4f84657504d71f867f91fbe7684eb48b4ab4aaf3612ae: Status 404 returned error can't find the container with id 651c4057a973ae689ee4f84657504d71f867f91fbe7684eb48b4ab4aaf3612ae Jul 30 20:26:09.478907 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.478890 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 20:26:09.499324 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:09.499278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" event={"ID":"c98bb757-2b33-4ae5-944e-abc2a991b0df","Type":"ContainerStarted","Data":"651c4057a973ae689ee4f84657504d71f867f91fbe7684eb48b4ab4aaf3612ae"} Jul 30 20:26:12.037366 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:12.037327 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 20:26:12.037653 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:12.037399 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 20:26:12.517298 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:12.517264 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" event={"ID":"c98bb757-2b33-4ae5-944e-abc2a991b0df","Type":"ContainerStarted","Data":"a63bfed356a4534d45cb10c6df2e3c5bb13b9d8958aaeb2be627703e15880423"} Jul 30 20:26:12.517535 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:12.517366 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:26:12.543235 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:12.543178 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" podStartSLOduration=1.985097962 podStartE2EDuration="4.543158762s" totalImagesPullingTime="2.5580608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:26:12.541492101 +0000 UTC m=+311.006630460" watchObservedRunningTime="2026-07-30 20:26:12.543158762 +0000 UTC m=+311.008297109" Jul 30 20:26:13.524860 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:26:13.524832 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-54ds6" Jul 30 20:27:06.476423 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.476324 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79"] Jul 30 20:27:06.480176 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.480148 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.482969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.482938 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 20:27:06.482969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.482945 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 20:27:06.483833 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.483818 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-57lfn\"" Jul 30 20:27:06.488750 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.488728 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79"] Jul 30 20:27:06.513667 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.513635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.513832 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.513743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.513832 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.513786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdkv\" (UniqueName: \"kubernetes.io/projected/ea0f001c-d192-4630-a2fc-af32cb2e6e48-kube-api-access-7qdkv\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.587970 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.587933 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f"] Jul 30 20:27:06.591444 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.591424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.601009 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.600983 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f"] Jul 30 20:27:06.614549 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.614518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdkv\" (UniqueName: \"kubernetes.io/projected/ea0f001c-d192-4630-a2fc-af32cb2e6e48-kube-api-access-7qdkv\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.614728 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.614554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.614728 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.614581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.614728 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.614707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.614910 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.614748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9t7z\" (UniqueName: \"kubernetes.io/projected/8d338f2f-a948-4000-bb3d-54c3649eb0f6-kube-api-access-c9t7z\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.614910 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.614839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.614978 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.614918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.615157 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.615138 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.623392 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.623363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdkv\" (UniqueName: \"kubernetes.io/projected/ea0f001c-d192-4630-a2fc-af32cb2e6e48-kube-api-access-7qdkv\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.674409 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.674371 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb"] Jul 30 20:27:06.678186 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.678165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.687268 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.687239 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb"] Jul 30 20:27:06.715264 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.715226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.715264 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.715267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.715528 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.715314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.715528 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.715360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.715528 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.715406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9t7z\" (UniqueName: \"kubernetes.io/projected/8d338f2f-a948-4000-bb3d-54c3649eb0f6-kube-api-access-c9t7z\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.715528 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.715458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6ml\" (UniqueName: \"kubernetes.io/projected/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-kube-api-access-jc6ml\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.715758 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.715689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.715758 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.715705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.724464 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.724431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9t7z\" (UniqueName: \"kubernetes.io/projected/8d338f2f-a948-4000-bb3d-54c3649eb0f6-kube-api-access-c9t7z\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.789879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.789790 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:06.809401 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.809366 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79"] Jul 30 20:27:06.813860 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.813835 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.815934 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.815902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.816105 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.815943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.816105 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.816073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6ml\" (UniqueName: \"kubernetes.io/projected/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-kube-api-access-jc6ml\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.816242 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.816135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.816242 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.816162 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j4ds\" (UniqueName: \"kubernetes.io/projected/93ea0d80-6bd3-408a-b788-e8ce32813fef-kube-api-access-9j4ds\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.816353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.816246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.816412 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.816374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.816564 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.816545 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.834447 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.834403 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79"] Jul 30 20:27:06.834897 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.834873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6ml\" (UniqueName: \"kubernetes.io/projected/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-kube-api-access-jc6ml\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:06.901404 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.901364 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:06.917523 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.917489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.917523 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.917529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j4ds\" (UniqueName: \"kubernetes.io/projected/93ea0d80-6bd3-408a-b788-e8ce32813fef-kube-api-access-9j4ds\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.917724 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.917591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.918024 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.918001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.918129 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.918001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.930494 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.930462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j4ds\" (UniqueName: \"kubernetes.io/projected/93ea0d80-6bd3-408a-b788-e8ce32813fef-kube-api-access-9j4ds\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:06.945498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.945462 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79"] Jul 30 20:27:06.947438 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:27:06.947400 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0f001c_d192_4630_a2fc_af32cb2e6e48.slice/crio-b3b2d6abb27d5657770c8def58b5593fd2958b970dd748c8f3caf5b1b4891c17 WatchSource:0}: Error finding container b3b2d6abb27d5657770c8def58b5593fd2958b970dd748c8f3caf5b1b4891c17: Status 404 returned error can't find the container with id b3b2d6abb27d5657770c8def58b5593fd2958b970dd748c8f3caf5b1b4891c17 Jul 30 20:27:06.988530 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:06.988495 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:07.071011 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.070928 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f"] Jul 30 20:27:07.071584 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:27:07.071529 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d338f2f_a948_4000_bb3d_54c3649eb0f6.slice/crio-5db7df675fa782e26e593b31c3c7b13756d4c199338b4a74c2958096c7a9fb71 WatchSource:0}: Error finding container 5db7df675fa782e26e593b31c3c7b13756d4c199338b4a74c2958096c7a9fb71: Status 404 returned error can't find the container with id 5db7df675fa782e26e593b31c3c7b13756d4c199338b4a74c2958096c7a9fb71 Jul 30 20:27:07.125115 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.125073 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:07.177553 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.177510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb"] Jul 30 20:27:07.181523 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:27:07.181461 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3466dbbb_2902_4fc4_808c_c6fc9e0de16a.slice/crio-d396891816ac4558c024b164c4b5f0e18bdb969c004e1f72331e075ee1062972 WatchSource:0}: Error finding container d396891816ac4558c024b164c4b5f0e18bdb969c004e1f72331e075ee1062972: Status 404 returned error can't find the container with id d396891816ac4558c024b164c4b5f0e18bdb969c004e1f72331e075ee1062972 Jul 30 20:27:07.284435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.284401 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79"] Jul 30 20:27:07.287602 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:27:07.287569 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ea0d80_6bd3_408a_b788_e8ce32813fef.slice/crio-ceafa837a017ecbcd10d58eb4a6f5a917e7612a2e6b21f2a9dc858da49de7f6a WatchSource:0}: Error finding container ceafa837a017ecbcd10d58eb4a6f5a917e7612a2e6b21f2a9dc858da49de7f6a: Status 404 returned error can't find the container with id ceafa837a017ecbcd10d58eb4a6f5a917e7612a2e6b21f2a9dc858da49de7f6a Jul 30 20:27:07.787325 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.787288 2575 generic.go:356] "Generic (PLEG): container finished" podID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerID="8fdd5994dd83e9bc62a5926bd14541aa3f96bf13604d9611b5039e4e135fcb49" exitCode=0 Jul 30 20:27:07.787736 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.787386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" event={"ID":"93ea0d80-6bd3-408a-b788-e8ce32813fef","Type":"ContainerDied","Data":"8fdd5994dd83e9bc62a5926bd14541aa3f96bf13604d9611b5039e4e135fcb49"} Jul 30 20:27:07.787736 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.787449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" event={"ID":"93ea0d80-6bd3-408a-b788-e8ce32813fef","Type":"ContainerStarted","Data":"ceafa837a017ecbcd10d58eb4a6f5a917e7612a2e6b21f2a9dc858da49de7f6a"} Jul 30 20:27:07.788915 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.788885 2575 generic.go:356] "Generic (PLEG): container finished" podID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerID="ff44ef4fca029327bd134e7bb27464b28fdc0fecc1613423c0f3f223c7d548d9" exitCode=0 Jul 30 20:27:07.788992 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.788972 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" event={"ID":"3466dbbb-2902-4fc4-808c-c6fc9e0de16a","Type":"ContainerDied","Data":"ff44ef4fca029327bd134e7bb27464b28fdc0fecc1613423c0f3f223c7d548d9"} Jul 30 20:27:07.789046 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.789021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" event={"ID":"3466dbbb-2902-4fc4-808c-c6fc9e0de16a","Type":"ContainerStarted","Data":"d396891816ac4558c024b164c4b5f0e18bdb969c004e1f72331e075ee1062972"} Jul 30 20:27:07.790596 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.790527 2575 generic.go:356] "Generic (PLEG): container finished" podID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerID="35e6916250548fadcf74e86b297443e19f08e1c0b6056f727b3d633e64d7a88f" exitCode=0 Jul 30 20:27:07.790662 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.790630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" event={"ID":"ea0f001c-d192-4630-a2fc-af32cb2e6e48","Type":"ContainerDied","Data":"35e6916250548fadcf74e86b297443e19f08e1c0b6056f727b3d633e64d7a88f"} Jul 30 20:27:07.790717 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.790678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" event={"ID":"ea0f001c-d192-4630-a2fc-af32cb2e6e48","Type":"ContainerStarted","Data":"b3b2d6abb27d5657770c8def58b5593fd2958b970dd748c8f3caf5b1b4891c17"} Jul 30 20:27:07.792284 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.792263 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerID="0147c8dca35e8f7765772ae67b24a5d716c1ddfc41552970d48cacf385e846a3" exitCode=0 Jul 30 20:27:07.792367 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.792336 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" event={"ID":"8d338f2f-a948-4000-bb3d-54c3649eb0f6","Type":"ContainerDied","Data":"0147c8dca35e8f7765772ae67b24a5d716c1ddfc41552970d48cacf385e846a3"} Jul 30 20:27:07.792441 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:07.792386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" event={"ID":"8d338f2f-a948-4000-bb3d-54c3649eb0f6","Type":"ContainerStarted","Data":"5db7df675fa782e26e593b31c3c7b13756d4c199338b4a74c2958096c7a9fb71"} Jul 30 20:27:09.805039 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:09.805004 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerID="d9958056bc9d296effca599aeab6bef4ed7db431b5ace0984b5f8e5ed86d07aa" exitCode=0 Jul 30 20:27:09.805481 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:09.805097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" event={"ID":"8d338f2f-a948-4000-bb3d-54c3649eb0f6","Type":"ContainerDied","Data":"d9958056bc9d296effca599aeab6bef4ed7db431b5ace0984b5f8e5ed86d07aa"} Jul 30 20:27:09.807192 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:09.807166 2575 generic.go:356] "Generic (PLEG): container finished" podID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerID="17dc787b062701fcb887640e6092eb337bcde6f622c114ceb6cd99c7c622a189" exitCode=0 Jul 30 20:27:09.807281 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:09.807250 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" event={"ID":"3466dbbb-2902-4fc4-808c-c6fc9e0de16a","Type":"ContainerDied","Data":"17dc787b062701fcb887640e6092eb337bcde6f622c114ceb6cd99c7c622a189"} Jul 30 20:27:09.809244 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:09.809219 2575 generic.go:356] "Generic (PLEG): container finished" podID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerID="69957a9b91e949e215e1044378714bb5c87168c1ea9e47a66fba6ec9dd3f315f" exitCode=0 Jul 30 20:27:09.809338 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:09.809246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" event={"ID":"ea0f001c-d192-4630-a2fc-af32cb2e6e48","Type":"ContainerDied","Data":"69957a9b91e949e215e1044378714bb5c87168c1ea9e47a66fba6ec9dd3f315f"} Jul 30 20:27:10.816864 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:10.816828 2575 generic.go:356] "Generic (PLEG): container finished" podID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerID="75158370c3796cd718eaadc7cf70dda09e1fb306048772ac342bb7c93c928de8" exitCode=0 Jul 30 20:27:10.817318 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:10.816907 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" event={"ID":"3466dbbb-2902-4fc4-808c-c6fc9e0de16a","Type":"ContainerDied","Data":"75158370c3796cd718eaadc7cf70dda09e1fb306048772ac342bb7c93c928de8"} Jul 30 20:27:10.818909 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:10.818880 2575 generic.go:356] "Generic (PLEG): container finished" podID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerID="31f6f4f33ef04d002222bbc38c361986f20f28a0332c1cb5d6caa4fd8db6d67e" exitCode=0 Jul 30 20:27:10.819037 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:10.818947 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" event={"ID":"ea0f001c-d192-4630-a2fc-af32cb2e6e48","Type":"ContainerDied","Data":"31f6f4f33ef04d002222bbc38c361986f20f28a0332c1cb5d6caa4fd8db6d67e"} Jul 30 20:27:10.820891 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:10.820866 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerID="8c21b481d2f256f4a70c13d63e54a204df145c58ee57a900b634bd914be4688d" exitCode=0 Jul 30 20:27:10.820999 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:10.820926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" event={"ID":"8d338f2f-a948-4000-bb3d-54c3649eb0f6","Type":"ContainerDied","Data":"8c21b481d2f256f4a70c13d63e54a204df145c58ee57a900b634bd914be4688d"} Jul 30 20:27:11.985230 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:11.985198 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:12.029861 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.029832 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:12.032399 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.032376 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:12.058638 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058611 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-bundle\") pod \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " Jul 30 20:27:12.058879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058657 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-bundle\") pod \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " Jul 30 20:27:12.058879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058677 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qdkv\" (UniqueName: \"kubernetes.io/projected/ea0f001c-d192-4630-a2fc-af32cb2e6e48-kube-api-access-7qdkv\") pod \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " Jul 30 20:27:12.058879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058715 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc6ml\" (UniqueName: \"kubernetes.io/projected/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-kube-api-access-jc6ml\") pod \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " Jul 30 20:27:12.058879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058738 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-util\") pod \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " Jul 30 20:27:12.058879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058762 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-bundle\") pod \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\" (UID: \"3466dbbb-2902-4fc4-808c-c6fc9e0de16a\") " Jul 30 20:27:12.058879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058796 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-util\") pod \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\" (UID: \"ea0f001c-d192-4630-a2fc-af32cb2e6e48\") " Jul 30 20:27:12.058879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058820 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-util\") pod \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " Jul 30 20:27:12.058879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.058842 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9t7z\" (UniqueName: \"kubernetes.io/projected/8d338f2f-a948-4000-bb3d-54c3649eb0f6-kube-api-access-c9t7z\") pod \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\" (UID: \"8d338f2f-a948-4000-bb3d-54c3649eb0f6\") " Jul 30 20:27:12.059711 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.059619 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-bundle" (OuterVolumeSpecName: "bundle") pod "ea0f001c-d192-4630-a2fc-af32cb2e6e48" (UID: "ea0f001c-d192-4630-a2fc-af32cb2e6e48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:12.059711 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.059619 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-bundle" (OuterVolumeSpecName: "bundle") pod "8d338f2f-a948-4000-bb3d-54c3649eb0f6" (UID: "8d338f2f-a948-4000-bb3d-54c3649eb0f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:12.060143 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.060118 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-bundle" (OuterVolumeSpecName: "bundle") pod "3466dbbb-2902-4fc4-808c-c6fc9e0de16a" (UID: "3466dbbb-2902-4fc4-808c-c6fc9e0de16a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:12.061737 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.061698 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0f001c-d192-4630-a2fc-af32cb2e6e48-kube-api-access-7qdkv" (OuterVolumeSpecName: "kube-api-access-7qdkv") pod "ea0f001c-d192-4630-a2fc-af32cb2e6e48" (UID: "ea0f001c-d192-4630-a2fc-af32cb2e6e48"). InnerVolumeSpecName "kube-api-access-7qdkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:12.062347 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.062321 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-kube-api-access-jc6ml" (OuterVolumeSpecName: "kube-api-access-jc6ml") pod "3466dbbb-2902-4fc4-808c-c6fc9e0de16a" (UID: "3466dbbb-2902-4fc4-808c-c6fc9e0de16a"). InnerVolumeSpecName "kube-api-access-jc6ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:12.062543 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.062519 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d338f2f-a948-4000-bb3d-54c3649eb0f6-kube-api-access-c9t7z" (OuterVolumeSpecName: "kube-api-access-c9t7z") pod "8d338f2f-a948-4000-bb3d-54c3649eb0f6" (UID: "8d338f2f-a948-4000-bb3d-54c3649eb0f6"). InnerVolumeSpecName "kube-api-access-c9t7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:12.067155 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.067121 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-util" (OuterVolumeSpecName: "util") pod "8d338f2f-a948-4000-bb3d-54c3649eb0f6" (UID: "8d338f2f-a948-4000-bb3d-54c3649eb0f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:12.068558 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.068510 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-util" (OuterVolumeSpecName: "util") pod "ea0f001c-d192-4630-a2fc-af32cb2e6e48" (UID: "ea0f001c-d192-4630-a2fc-af32cb2e6e48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:12.070012 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.069960 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-util" (OuterVolumeSpecName: "util") pod "3466dbbb-2902-4fc4-808c-c6fc9e0de16a" (UID: "3466dbbb-2902-4fc4-808c-c6fc9e0de16a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159740 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6ml\" (UniqueName: \"kubernetes.io/projected/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-kube-api-access-jc6ml\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159774 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-util\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159785 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3466dbbb-2902-4fc4-808c-c6fc9e0de16a-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159793 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-util\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159802 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-util\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159810 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9t7z\" (UniqueName: \"kubernetes.io/projected/8d338f2f-a948-4000-bb3d-54c3649eb0f6-kube-api-access-c9t7z\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159819 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea0f001c-d192-4630-a2fc-af32cb2e6e48-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159828 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d338f2f-a948-4000-bb3d-54c3649eb0f6-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.159836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.159836 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdkv\" (UniqueName: \"kubernetes.io/projected/ea0f001c-d192-4630-a2fc-af32cb2e6e48-kube-api-access-7qdkv\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:12.832299 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.832268 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" Jul 30 20:27:12.832498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.832266 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05tncb" event={"ID":"3466dbbb-2902-4fc4-808c-c6fc9e0de16a","Type":"ContainerDied","Data":"d396891816ac4558c024b164c4b5f0e18bdb969c004e1f72331e075ee1062972"} Jul 30 20:27:12.832498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.832387 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d396891816ac4558c024b164c4b5f0e18bdb969c004e1f72331e075ee1062972" Jul 30 20:27:12.833988 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.833961 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" Jul 30 20:27:12.834155 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.833981 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535pk79" event={"ID":"ea0f001c-d192-4630-a2fc-af32cb2e6e48","Type":"ContainerDied","Data":"b3b2d6abb27d5657770c8def58b5593fd2958b970dd748c8f3caf5b1b4891c17"} Jul 30 20:27:12.834155 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.834033 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b2d6abb27d5657770c8def58b5593fd2958b970dd748c8f3caf5b1b4891c17" Jul 30 20:27:12.836168 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.836147 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" Jul 30 20:27:12.836168 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.836150 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c25v9f" event={"ID":"8d338f2f-a948-4000-bb3d-54c3649eb0f6","Type":"ContainerDied","Data":"5db7df675fa782e26e593b31c3c7b13756d4c199338b4a74c2958096c7a9fb71"} Jul 30 20:27:12.836315 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:12.836193 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db7df675fa782e26e593b31c3c7b13756d4c199338b4a74c2958096c7a9fb71" Jul 30 20:27:13.843007 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:13.842972 2575 generic.go:356] "Generic (PLEG): container finished" podID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerID="b466386dd9759c4044e37bb6ce0a6be872d00f38b3e0f6056d8b275eb6af4980" exitCode=0 Jul 30 20:27:13.843535 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:13.843072 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" event={"ID":"93ea0d80-6bd3-408a-b788-e8ce32813fef","Type":"ContainerDied","Data":"b466386dd9759c4044e37bb6ce0a6be872d00f38b3e0f6056d8b275eb6af4980"} Jul 30 20:27:14.849389 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:14.849356 2575 generic.go:356] "Generic (PLEG): container finished" podID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerID="b48fad7275c566a622dbb3a932dc764bc7eca5cf75c4e463d709c620e6044118" exitCode=0 Jul 30 20:27:14.849814 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:14.849439 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" event={"ID":"93ea0d80-6bd3-408a-b788-e8ce32813fef","Type":"ContainerDied","Data":"b48fad7275c566a622dbb3a932dc764bc7eca5cf75c4e463d709c620e6044118"} Jul 30 20:27:15.988326 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:15.988295 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:16.094172 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.094134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-bundle\") pod \"93ea0d80-6bd3-408a-b788-e8ce32813fef\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " Jul 30 20:27:16.094347 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.094204 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j4ds\" (UniqueName: \"kubernetes.io/projected/93ea0d80-6bd3-408a-b788-e8ce32813fef-kube-api-access-9j4ds\") pod \"93ea0d80-6bd3-408a-b788-e8ce32813fef\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " Jul 30 20:27:16.094347 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.094263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-util\") pod \"93ea0d80-6bd3-408a-b788-e8ce32813fef\" (UID: \"93ea0d80-6bd3-408a-b788-e8ce32813fef\") " Jul 30 20:27:16.094775 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.094738 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-bundle" (OuterVolumeSpecName: "bundle") pod "93ea0d80-6bd3-408a-b788-e8ce32813fef" (UID: "93ea0d80-6bd3-408a-b788-e8ce32813fef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:16.096327 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.096302 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ea0d80-6bd3-408a-b788-e8ce32813fef-kube-api-access-9j4ds" (OuterVolumeSpecName: "kube-api-access-9j4ds") pod "93ea0d80-6bd3-408a-b788-e8ce32813fef" (UID: "93ea0d80-6bd3-408a-b788-e8ce32813fef"). InnerVolumeSpecName "kube-api-access-9j4ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:16.099246 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.099220 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-util" (OuterVolumeSpecName: "util") pod "93ea0d80-6bd3-408a-b788-e8ce32813fef" (UID: "93ea0d80-6bd3-408a-b788-e8ce32813fef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:16.195471 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.195369 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:16.195471 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.195410 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j4ds\" (UniqueName: \"kubernetes.io/projected/93ea0d80-6bd3-408a-b788-e8ce32813fef-kube-api-access-9j4ds\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:16.195471 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.195423 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93ea0d80-6bd3-408a-b788-e8ce32813fef-util\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:16.862599 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.862568 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" Jul 30 20:27:16.862780 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.862574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbg7z79" event={"ID":"93ea0d80-6bd3-408a-b788-e8ce32813fef","Type":"ContainerDied","Data":"ceafa837a017ecbcd10d58eb4a6f5a917e7612a2e6b21f2a9dc858da49de7f6a"} Jul 30 20:27:16.862780 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:16.862684 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceafa837a017ecbcd10d58eb4a6f5a917e7612a2e6b21f2a9dc858da49de7f6a" Jul 30 20:27:32.004947 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.004913 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b7d76c748-f7sk8"] Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005304 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerName="pull" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005321 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerName="pull" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005333 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerName="util" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005339 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerName="util" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005347 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerName="extract" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005353 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerName="extract" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005362 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerName="extract" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005367 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerName="extract" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005375 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerName="util" Jul 30 20:27:32.005377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005379 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerName="util" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005388 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerName="pull" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005393 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerName="pull" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005401 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerName="pull" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005407 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerName="pull" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005415 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerName="pull" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005420 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerName="pull" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005428 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerName="extract" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005432 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerName="extract" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005439 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerName="util" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005444 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerName="util" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005452 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerName="util" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005457 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerName="util" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005463 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerName="extract" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005468 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerName="extract" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005519 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ea0d80-6bd3-408a-b788-e8ce32813fef" containerName="extract" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005526 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0f001c-d192-4630-a2fc-af32cb2e6e48" containerName="extract" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005534 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3466dbbb-2902-4fc4-808c-c6fc9e0de16a" containerName="extract" Jul 30 20:27:32.005768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.005540 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d338f2f-a948-4000-bb3d-54c3649eb0f6" containerName="extract" Jul 30 20:27:32.013176 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.013150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.020844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.020801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7d76c748-f7sk8"] Jul 30 20:27:32.125222 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.125181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-oauth-serving-cert\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.125222 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.125222 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-config\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.125498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.125239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-serving-cert\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.125498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.125301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-service-ca\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.125498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.125350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-trusted-ca-bundle\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.125498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.125392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-oauth-config\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.125498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.125455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxfk\" (UniqueName: \"kubernetes.io/projected/aafaa4d9-0f42-4436-9ce9-48e328d866b8-kube-api-access-mnxfk\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.226646 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.226608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-oauth-config\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.226817 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.226704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxfk\" (UniqueName: \"kubernetes.io/projected/aafaa4d9-0f42-4436-9ce9-48e328d866b8-kube-api-access-mnxfk\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.226817 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.226774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-oauth-serving-cert\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.226817 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.226802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-config\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.226954 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.226831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-serving-cert\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.226954 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.226860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-service-ca\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.226954 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.226895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-trusted-ca-bundle\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.227624 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.227596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-oauth-serving-cert\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.227733 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.227689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-service-ca\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.227893 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.227847 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-config\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.228022 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.228001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aafaa4d9-0f42-4436-9ce9-48e328d866b8-trusted-ca-bundle\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.229766 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.229745 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-oauth-config\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.229857 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.229804 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aafaa4d9-0f42-4436-9ce9-48e328d866b8-console-serving-cert\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.239976 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.239957 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxfk\" (UniqueName: \"kubernetes.io/projected/aafaa4d9-0f42-4436-9ce9-48e328d866b8-kube-api-access-mnxfk\") pod \"console-7b7d76c748-f7sk8\" (UID: \"aafaa4d9-0f42-4436-9ce9-48e328d866b8\") " pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.326025 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.325934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:32.474239 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.474211 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7d76c748-f7sk8"] Jul 30 20:27:32.476304 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:27:32.476271 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafaa4d9_0f42_4436_9ce9_48e328d866b8.slice/crio-b2cbe0ee96dc38ad1c2a9a1a3912e28706a5286753db8888690ff53e687f144d WatchSource:0}: Error finding container b2cbe0ee96dc38ad1c2a9a1a3912e28706a5286753db8888690ff53e687f144d: Status 404 returned error can't find the container with id b2cbe0ee96dc38ad1c2a9a1a3912e28706a5286753db8888690ff53e687f144d Jul 30 20:27:32.947460 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.947419 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7d76c748-f7sk8" event={"ID":"aafaa4d9-0f42-4436-9ce9-48e328d866b8","Type":"ContainerStarted","Data":"71cfeb916deabf16a0535b5dec66f2ea67da10f91708d295e12049b531a319b6"} Jul 30 20:27:32.947670 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.947484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7d76c748-f7sk8" event={"ID":"aafaa4d9-0f42-4436-9ce9-48e328d866b8","Type":"ContainerStarted","Data":"b2cbe0ee96dc38ad1c2a9a1a3912e28706a5286753db8888690ff53e687f144d"} Jul 30 20:27:32.973471 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:32.973417 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b7d76c748-f7sk8" podStartSLOduration=1.9734017179999999 podStartE2EDuration="1.973401718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:27:32.971895346 +0000 UTC m=+391.437033697" watchObservedRunningTime="2026-07-30 20:27:32.973401718 +0000 UTC m=+391.438540065" Jul 30 20:27:42.326688 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:42.326641 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:42.327175 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:42.326726 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:42.331622 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:42.331599 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:43.002580 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:43.002551 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b7d76c748-f7sk8" Jul 30 20:27:43.095323 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:43.095293 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678c664c85-ghk9z"] Jul 30 20:27:44.472482 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.472448 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj"] Jul 30 20:27:44.476355 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.476330 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:44.479418 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.479153 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 20:27:44.479810 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.479509 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-56ccm\"" Jul 30 20:27:44.479984 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.479928 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 20:27:44.492143 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.492098 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj"] Jul 30 20:27:44.535267 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.535229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe7acf02-4b46-4bc3-80a3-23dffc96b700-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7cshj\" (UID: \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:44.535463 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.535328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lbdl\" (UniqueName: \"kubernetes.io/projected/fe7acf02-4b46-4bc3-80a3-23dffc96b700-kube-api-access-9lbdl\") pod \"kuadrant-operator-controller-manager-58fc56757c-7cshj\" (UID: \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:44.636467 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.636422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe7acf02-4b46-4bc3-80a3-23dffc96b700-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7cshj\" (UID: \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:44.636647 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.636479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lbdl\" (UniqueName: \"kubernetes.io/projected/fe7acf02-4b46-4bc3-80a3-23dffc96b700-kube-api-access-9lbdl\") pod \"kuadrant-operator-controller-manager-58fc56757c-7cshj\" (UID: \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:44.636786 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.636767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe7acf02-4b46-4bc3-80a3-23dffc96b700-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7cshj\" (UID: \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:44.646549 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.646514 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lbdl\" (UniqueName: \"kubernetes.io/projected/fe7acf02-4b46-4bc3-80a3-23dffc96b700-kube-api-access-9lbdl\") pod \"kuadrant-operator-controller-manager-58fc56757c-7cshj\" (UID: \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:44.789664 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.789565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:44.948282 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:44.948255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj"] Jul 30 20:27:44.951234 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:27:44.951209 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7acf02_4b46_4bc3_80a3_23dffc96b700.slice/crio-72433b55b458f97da9dd6a01d077dc911c2f52de996b81f2e5acccee9ff349ac WatchSource:0}: Error finding container 72433b55b458f97da9dd6a01d077dc911c2f52de996b81f2e5acccee9ff349ac: Status 404 returned error can't find the container with id 72433b55b458f97da9dd6a01d077dc911c2f52de996b81f2e5acccee9ff349ac Jul 30 20:27:45.009813 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:45.009780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" event={"ID":"fe7acf02-4b46-4bc3-80a3-23dffc96b700","Type":"ContainerStarted","Data":"72433b55b458f97da9dd6a01d077dc911c2f52de996b81f2e5acccee9ff349ac"} Jul 30 20:27:49.038724 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:49.038688 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" event={"ID":"fe7acf02-4b46-4bc3-80a3-23dffc96b700","Type":"ContainerStarted","Data":"d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f"} Jul 30 20:27:49.039185 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:49.038868 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:49.066038 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:49.065982 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" podStartSLOduration=1.181319256 podStartE2EDuration="5.065964417s" totalImagesPullingTime="3.884645161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:27:49.063411143 +0000 UTC m=+407.528549492" watchObservedRunningTime="2026-07-30 20:27:49.065964417 +0000 UTC m=+407.531102764" Jul 30 20:27:56.728466 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:56.728424 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj"] Jul 30 20:27:56.729037 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:56.728695 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" podUID="fe7acf02-4b46-4bc3-80a3-23dffc96b700" containerName="manager" containerID="cri-o://d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f" gracePeriod=10 Jul 30 20:27:56.730578 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:56.730551 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:56.980185 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:56.980114 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:57.083715 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.083680 2575 generic.go:356] "Generic (PLEG): container finished" podID="fe7acf02-4b46-4bc3-80a3-23dffc96b700" containerID="d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f" exitCode=0 Jul 30 20:27:57.083915 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.083727 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" Jul 30 20:27:57.083915 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.083770 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" event={"ID":"fe7acf02-4b46-4bc3-80a3-23dffc96b700","Type":"ContainerDied","Data":"d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f"} Jul 30 20:27:57.083915 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.083832 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj" event={"ID":"fe7acf02-4b46-4bc3-80a3-23dffc96b700","Type":"ContainerDied","Data":"72433b55b458f97da9dd6a01d077dc911c2f52de996b81f2e5acccee9ff349ac"} Jul 30 20:27:57.083915 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.083856 2575 scope.go:117] "RemoveContainer" containerID="d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f" Jul 30 20:27:57.093308 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.093283 2575 scope.go:117] "RemoveContainer" containerID="d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f" Jul 30 20:27:57.093686 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:27:57.093660 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f\": container with ID starting with d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f not found: ID does not exist" containerID="d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f" Jul 30 20:27:57.093750 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.093698 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f"} err="failed to get container status \"d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f\": rpc error: code = NotFound desc = could not find container \"d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f\": container with ID starting with d5a74b5862767172d6f55eba9a9f305e7955e163c50642b1a616eebfca848a2f not found: ID does not exist" Jul 30 20:27:57.151686 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.151655 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe7acf02-4b46-4bc3-80a3-23dffc96b700-extensions-socket-volume\") pod \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\" (UID: \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\") " Jul 30 20:27:57.151869 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.151702 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lbdl\" (UniqueName: \"kubernetes.io/projected/fe7acf02-4b46-4bc3-80a3-23dffc96b700-kube-api-access-9lbdl\") pod \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\" (UID: \"fe7acf02-4b46-4bc3-80a3-23dffc96b700\") " Jul 30 20:27:57.152211 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.152181 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7acf02-4b46-4bc3-80a3-23dffc96b700-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fe7acf02-4b46-4bc3-80a3-23dffc96b700" (UID: "fe7acf02-4b46-4bc3-80a3-23dffc96b700"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:57.154141 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.154107 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7acf02-4b46-4bc3-80a3-23dffc96b700-kube-api-access-9lbdl" (OuterVolumeSpecName: "kube-api-access-9lbdl") pod "fe7acf02-4b46-4bc3-80a3-23dffc96b700" (UID: "fe7acf02-4b46-4bc3-80a3-23dffc96b700"). InnerVolumeSpecName "kube-api-access-9lbdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:57.252945 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.252850 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe7acf02-4b46-4bc3-80a3-23dffc96b700-extensions-socket-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:57.252945 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.252886 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lbdl\" (UniqueName: \"kubernetes.io/projected/fe7acf02-4b46-4bc3-80a3-23dffc96b700-kube-api-access-9lbdl\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:27:57.419812 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.419781 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj"] Jul 30 20:27:57.430966 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:57.430930 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7cshj"] Jul 30 20:27:58.012417 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:27:58.012374 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe7acf02-4b46-4bc3-80a3-23dffc96b700" path="/var/lib/kubelet/pods/fe7acf02-4b46-4bc3-80a3-23dffc96b700/volumes" Jul 30 20:28:08.115344 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.115273 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-678c664c85-ghk9z" podUID="b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" containerName="console" containerID="cri-o://359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6" gracePeriod=15 Jul 30 20:28:08.365731 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.365655 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:28:08.453723 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.453682 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-oauth-config\") pod \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " Jul 30 20:28:08.453723 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.453722 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-serving-cert\") pod \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " Jul 30 20:28:08.453942 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.453760 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-config\") pod \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " Jul 30 20:28:08.453942 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.453857 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj444\" (UniqueName: \"kubernetes.io/projected/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-kube-api-access-xj444\") pod \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " Jul 30 20:28:08.453942 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.453914 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-trusted-ca-bundle\") pod \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " Jul 30 20:28:08.454084 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.453977 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-service-ca\") pod \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " Jul 30 20:28:08.454084 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.454007 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-oauth-serving-cert\") pod \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\" (UID: \"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d\") " Jul 30 20:28:08.454329 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.454292 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-config" (OuterVolumeSpecName: "console-config") pod "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" (UID: "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:28:08.454457 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.454427 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-service-ca" (OuterVolumeSpecName: "service-ca") pod "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" (UID: "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:28:08.454556 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.454527 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" (UID: "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:28:08.454760 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.454737 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" (UID: "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:28:08.455945 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.455922 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" (UID: "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:28:08.456009 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.455966 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-kube-api-access-xj444" (OuterVolumeSpecName: "kube-api-access-xj444") pod "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" (UID: "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d"). InnerVolumeSpecName "kube-api-access-xj444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:28:08.456009 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.455980 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" (UID: "b86d9a9f-f7e8-4213-a86e-6cec533f6d3d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:28:08.554784 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.554741 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-trusted-ca-bundle\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:08.554784 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.554774 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:08.554784 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.554786 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-oauth-serving-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:08.554784 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.554795 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-oauth-config\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:08.555096 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.554803 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-serving-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:08.555096 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.554813 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-console-config\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:08.555096 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:08.554821 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj444\" (UniqueName: \"kubernetes.io/projected/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d-kube-api-access-xj444\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:09.150302 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.150267 2575 generic.go:356] "Generic (PLEG): container finished" podID="b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" containerID="359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6" exitCode=0 Jul 30 20:28:09.150710 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.150328 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678c664c85-ghk9z" Jul 30 20:28:09.150710 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.150331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678c664c85-ghk9z" event={"ID":"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d","Type":"ContainerDied","Data":"359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6"} Jul 30 20:28:09.150710 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.150489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678c664c85-ghk9z" event={"ID":"b86d9a9f-f7e8-4213-a86e-6cec533f6d3d","Type":"ContainerDied","Data":"ac180ac6ed01499d88d664fa73ec7bd648d5d168e438b54978e46842500d06e0"} Jul 30 20:28:09.150710 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.150520 2575 scope.go:117] "RemoveContainer" containerID="359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6" Jul 30 20:28:09.159195 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.159175 2575 scope.go:117] "RemoveContainer" containerID="359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6" Jul 30 20:28:09.159450 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:28:09.159430 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6\": container with ID starting with 359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6 not found: ID does not exist" containerID="359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6" Jul 30 20:28:09.159503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.159458 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6"} err="failed to get container status \"359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6\": rpc error: code = NotFound desc = could not find container \"359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6\": container with ID starting with 359b1f4b093e57b834c2933965102de24a3d19cadbaef8314ee08733a53b0db6 not found: ID does not exist" Jul 30 20:28:09.178884 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.178858 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678c664c85-ghk9z"] Jul 30 20:28:09.191044 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:09.191017 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-678c664c85-ghk9z"] Jul 30 20:28:10.012465 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:10.012423 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" path="/var/lib/kubelet/pods/b86d9a9f-f7e8-4213-a86e-6cec533f6d3d/volumes" Jul 30 20:28:14.916022 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.915987 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-rl6fj"] Jul 30 20:28:14.916468 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.916334 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" containerName="console" Jul 30 20:28:14.916468 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.916345 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" containerName="console" Jul 30 20:28:14.916468 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.916356 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe7acf02-4b46-4bc3-80a3-23dffc96b700" containerName="manager" Jul 30 20:28:14.916468 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.916362 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe7acf02-4b46-4bc3-80a3-23dffc96b700" containerName="manager" Jul 30 20:28:14.916468 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.916416 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86d9a9f-f7e8-4213-a86e-6cec533f6d3d" containerName="console" Jul 30 20:28:14.916468 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.916426 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe7acf02-4b46-4bc3-80a3-23dffc96b700" containerName="manager" Jul 30 20:28:14.920966 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.920947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:14.923724 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.923692 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 20:28:14.923724 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.923721 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 30 20:28:14.925028 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.925000 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 20:28:14.925028 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.925020 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7ctr8\"" Jul 30 20:28:14.928348 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:14.928226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-rl6fj"] Jul 30 20:28:15.006414 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.006377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jzn\" (UniqueName: \"kubernetes.io/projected/036e87e7-5548-488b-8e36-d9e31c3ec494-kube-api-access-m8jzn\") pod \"limitador-limitador-859c4d96d6-rl6fj\" (UID: \"036e87e7-5548-488b-8e36-d9e31c3ec494\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:15.006618 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.006530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/036e87e7-5548-488b-8e36-d9e31c3ec494-config-file\") pod \"limitador-limitador-859c4d96d6-rl6fj\" (UID: \"036e87e7-5548-488b-8e36-d9e31c3ec494\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:15.013924 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.013886 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-rl6fj"] Jul 30 20:28:15.107860 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.107820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jzn\" (UniqueName: \"kubernetes.io/projected/036e87e7-5548-488b-8e36-d9e31c3ec494-kube-api-access-m8jzn\") pod \"limitador-limitador-859c4d96d6-rl6fj\" (UID: \"036e87e7-5548-488b-8e36-d9e31c3ec494\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:15.108035 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.107967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/036e87e7-5548-488b-8e36-d9e31c3ec494-config-file\") pod \"limitador-limitador-859c4d96d6-rl6fj\" (UID: \"036e87e7-5548-488b-8e36-d9e31c3ec494\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:15.108564 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.108543 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/036e87e7-5548-488b-8e36-d9e31c3ec494-config-file\") pod \"limitador-limitador-859c4d96d6-rl6fj\" (UID: \"036e87e7-5548-488b-8e36-d9e31c3ec494\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:15.130354 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.130322 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jzn\" (UniqueName: \"kubernetes.io/projected/036e87e7-5548-488b-8e36-d9e31c3ec494-kube-api-access-m8jzn\") pod \"limitador-limitador-859c4d96d6-rl6fj\" (UID: \"036e87e7-5548-488b-8e36-d9e31c3ec494\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:15.232169 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.232124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:15.370072 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:15.370031 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-rl6fj"] Jul 30 20:28:15.372384 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:28:15.372348 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036e87e7_5548_488b_8e36_d9e31c3ec494.slice/crio-b0c39fea54ef33a8f3f37974897c5b4f56296c3150d222d3e1674a2709b4f2d5 WatchSource:0}: Error finding container b0c39fea54ef33a8f3f37974897c5b4f56296c3150d222d3e1674a2709b4f2d5: Status 404 returned error can't find the container with id b0c39fea54ef33a8f3f37974897c5b4f56296c3150d222d3e1674a2709b4f2d5 Jul 30 20:28:16.193545 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:16.193501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" event={"ID":"036e87e7-5548-488b-8e36-d9e31c3ec494","Type":"ContainerStarted","Data":"b0c39fea54ef33a8f3f37974897c5b4f56296c3150d222d3e1674a2709b4f2d5"} Jul 30 20:28:20.215044 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:20.215006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" event={"ID":"036e87e7-5548-488b-8e36-d9e31c3ec494","Type":"ContainerStarted","Data":"b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8"} Jul 30 20:28:20.215496 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:20.215092 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:20.235546 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:20.235494 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" podStartSLOduration=1.53733927 podStartE2EDuration="6.235477603s" totalImagesPullingTime="4.698138333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:28:20.234199709 +0000 UTC m=+438.699338057" watchObservedRunningTime="2026-07-30 20:28:20.235477603 +0000 UTC m=+438.700615950" Jul 30 20:28:28.874616 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:28.874531 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-rl6fj"] Jul 30 20:28:28.875353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:28.875322 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" podUID="036e87e7-5548-488b-8e36-d9e31c3ec494" containerName="limitador" containerID="cri-o://b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8" gracePeriod=30 Jul 30 20:28:28.876090 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:28.875973 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:29.420439 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:29.420409 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:29.533262 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:29.533228 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/036e87e7-5548-488b-8e36-d9e31c3ec494-config-file\") pod \"036e87e7-5548-488b-8e36-d9e31c3ec494\" (UID: \"036e87e7-5548-488b-8e36-d9e31c3ec494\") " Jul 30 20:28:29.533262 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:29.533267 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8jzn\" (UniqueName: \"kubernetes.io/projected/036e87e7-5548-488b-8e36-d9e31c3ec494-kube-api-access-m8jzn\") pod \"036e87e7-5548-488b-8e36-d9e31c3ec494\" (UID: \"036e87e7-5548-488b-8e36-d9e31c3ec494\") " Jul 30 20:28:29.533664 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:29.533637 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036e87e7-5548-488b-8e36-d9e31c3ec494-config-file" (OuterVolumeSpecName: "config-file") pod "036e87e7-5548-488b-8e36-d9e31c3ec494" (UID: "036e87e7-5548-488b-8e36-d9e31c3ec494"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:28:29.535424 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:29.535397 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036e87e7-5548-488b-8e36-d9e31c3ec494-kube-api-access-m8jzn" (OuterVolumeSpecName: "kube-api-access-m8jzn") pod "036e87e7-5548-488b-8e36-d9e31c3ec494" (UID: "036e87e7-5548-488b-8e36-d9e31c3ec494"). InnerVolumeSpecName "kube-api-access-m8jzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:28:29.633849 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:29.633796 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/036e87e7-5548-488b-8e36-d9e31c3ec494-config-file\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:29.633849 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:29.633841 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8jzn\" (UniqueName: \"kubernetes.io/projected/036e87e7-5548-488b-8e36-d9e31c3ec494-kube-api-access-m8jzn\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:28:30.264319 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.264282 2575 generic.go:356] "Generic (PLEG): container finished" podID="036e87e7-5548-488b-8e36-d9e31c3ec494" containerID="b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8" exitCode=0 Jul 30 20:28:30.264712 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.264342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" event={"ID":"036e87e7-5548-488b-8e36-d9e31c3ec494","Type":"ContainerDied","Data":"b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8"} Jul 30 20:28:30.264712 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.264384 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" event={"ID":"036e87e7-5548-488b-8e36-d9e31c3ec494","Type":"ContainerDied","Data":"b0c39fea54ef33a8f3f37974897c5b4f56296c3150d222d3e1674a2709b4f2d5"} Jul 30 20:28:30.264712 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.264405 2575 scope.go:117] "RemoveContainer" containerID="b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8" Jul 30 20:28:30.264712 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.264343 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-rl6fj" Jul 30 20:28:30.272804 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.272785 2575 scope.go:117] "RemoveContainer" containerID="b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8" Jul 30 20:28:30.273100 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:28:30.273080 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8\": container with ID starting with b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8 not found: ID does not exist" containerID="b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8" Jul 30 20:28:30.273165 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.273108 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8"} err="failed to get container status \"b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8\": rpc error: code = NotFound desc = could not find container \"b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8\": container with ID starting with b1bb9d89181f51df5c95946da9e2ece5844c29d36503535cf44027c7865e4cd8 not found: ID does not exist" Jul 30 20:28:30.287780 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.287703 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-rl6fj"] Jul 30 20:28:30.291758 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:30.291730 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-rl6fj"] Jul 30 20:28:31.478606 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.478573 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w"] Jul 30 20:28:31.479076 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.478923 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="036e87e7-5548-488b-8e36-d9e31c3ec494" containerName="limitador" Jul 30 20:28:31.479076 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.478938 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="036e87e7-5548-488b-8e36-d9e31c3ec494" containerName="limitador" Jul 30 20:28:31.479076 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.478998 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="036e87e7-5548-488b-8e36-d9e31c3ec494" containerName="limitador" Jul 30 20:28:31.483196 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.483175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:31.486362 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.486341 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 20:28:31.487308 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.487285 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nmcqg\"" Jul 30 20:28:31.487435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.487311 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 30 20:28:31.487435 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.487346 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 30 20:28:31.499262 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.499238 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w"] Jul 30 20:28:31.552683 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.552637 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1d8662c-f616-4dac-9323-1baa4820c6c5-data\") pod \"postgres-69fb4594cf-9hs7w\" (UID: \"a1d8662c-f616-4dac-9323-1baa4820c6c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:31.552856 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.552746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwrf\" (UniqueName: \"kubernetes.io/projected/a1d8662c-f616-4dac-9323-1baa4820c6c5-kube-api-access-9qwrf\") pod \"postgres-69fb4594cf-9hs7w\" (UID: \"a1d8662c-f616-4dac-9323-1baa4820c6c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:31.653530 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.653486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwrf\" (UniqueName: \"kubernetes.io/projected/a1d8662c-f616-4dac-9323-1baa4820c6c5-kube-api-access-9qwrf\") pod \"postgres-69fb4594cf-9hs7w\" (UID: \"a1d8662c-f616-4dac-9323-1baa4820c6c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:31.653715 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.653550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1d8662c-f616-4dac-9323-1baa4820c6c5-data\") pod \"postgres-69fb4594cf-9hs7w\" (UID: \"a1d8662c-f616-4dac-9323-1baa4820c6c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:31.653969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.653950 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1d8662c-f616-4dac-9323-1baa4820c6c5-data\") pod \"postgres-69fb4594cf-9hs7w\" (UID: \"a1d8662c-f616-4dac-9323-1baa4820c6c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:31.663007 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.662978 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwrf\" (UniqueName: \"kubernetes.io/projected/a1d8662c-f616-4dac-9323-1baa4820c6c5-kube-api-access-9qwrf\") pod \"postgres-69fb4594cf-9hs7w\" (UID: \"a1d8662c-f616-4dac-9323-1baa4820c6c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:31.794672 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.794579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:31.930650 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:31.930624 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w"] Jul 30 20:28:31.932996 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:28:31.932970 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d8662c_f616_4dac_9323_1baa4820c6c5.slice/crio-534a103530d2ad184e93c6df9a13e6bed5e8e721fca2483b396deef6fb060d77 WatchSource:0}: Error finding container 534a103530d2ad184e93c6df9a13e6bed5e8e721fca2483b396deef6fb060d77: Status 404 returned error can't find the container with id 534a103530d2ad184e93c6df9a13e6bed5e8e721fca2483b396deef6fb060d77 Jul 30 20:28:32.012543 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:32.012510 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="036e87e7-5548-488b-8e36-d9e31c3ec494" path="/var/lib/kubelet/pods/036e87e7-5548-488b-8e36-d9e31c3ec494/volumes" Jul 30 20:28:32.276249 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:32.276213 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" event={"ID":"a1d8662c-f616-4dac-9323-1baa4820c6c5","Type":"ContainerStarted","Data":"534a103530d2ad184e93c6df9a13e6bed5e8e721fca2483b396deef6fb060d77"} Jul 30 20:28:37.308316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:37.308276 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" event={"ID":"a1d8662c-f616-4dac-9323-1baa4820c6c5","Type":"ContainerStarted","Data":"cfc9503d8766d5b859534d18ad3666fd44985fb32bf53da4e3de5cb0028e3c56"} Jul 30 20:28:37.308685 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:37.308384 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:37.326530 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:37.326483 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" podStartSLOduration=1.4282998120000001 podStartE2EDuration="6.326468719s" totalImagesPullingTime="4.898168907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:28:37.325338886 +0000 UTC m=+455.790477235" watchObservedRunningTime="2026-07-30 20:28:37.326468719 +0000 UTC m=+455.791607127" Jul 30 20:28:43.342157 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:43.342123 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hs7w" Jul 30 20:28:50.938377 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:50.938320 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f4c996689-xqs7f"] Jul 30 20:28:50.963281 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:50.963246 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f4c996689-xqs7f"] Jul 30 20:28:50.963451 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:50.963380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:50.966180 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:50.966154 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 20:28:50.966180 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:50.966172 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kgvj9\"" Jul 30 20:28:50.966394 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:50.966172 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 20:28:51.108860 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.108823 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-gjjfq"] Jul 30 20:28:51.112260 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.112243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:51.121764 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.121735 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-gjjfq"] Jul 30 20:28:51.127538 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.127507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert\") pod \"maas-controller-5f4c996689-xqs7f\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:51.127681 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.127631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqxq\" (UniqueName: \"kubernetes.io/projected/09a7609b-8c62-48f7-a1f3-ec487d428a76-kube-api-access-2hqxq\") pod \"maas-controller-5f4c996689-xqs7f\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:51.228100 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.228027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f502cc28-13e7-462f-b79f-7d6840514ff1-webhook-cert\") pod \"maas-controller-864fc8cdd8-gjjfq\" (UID: \"f502cc28-13e7-462f-b79f-7d6840514ff1\") " pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:51.228307 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.228120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert\") pod \"maas-controller-5f4c996689-xqs7f\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:51.228307 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.228165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqxq\" (UniqueName: \"kubernetes.io/projected/09a7609b-8c62-48f7-a1f3-ec487d428a76-kube-api-access-2hqxq\") pod \"maas-controller-5f4c996689-xqs7f\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:51.228307 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:28:51.228208 2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 30 20:28:51.228307 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:28:51.228281 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert podName:09a7609b-8c62-48f7-a1f3-ec487d428a76 nodeName:}" failed. No retries permitted until 2026-07-30 20:28:51.728262524 +0000 UTC m=+470.193400849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert") pod "maas-controller-5f4c996689-xqs7f" (UID: "09a7609b-8c62-48f7-a1f3-ec487d428a76") : secret "maas-controller-webhook-cert" not found Jul 30 20:28:51.228307 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.228210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q8lf\" (UniqueName: \"kubernetes.io/projected/f502cc28-13e7-462f-b79f-7d6840514ff1-kube-api-access-8q8lf\") pod \"maas-controller-864fc8cdd8-gjjfq\" (UID: \"f502cc28-13e7-462f-b79f-7d6840514ff1\") " pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:51.237117 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.237078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqxq\" (UniqueName: \"kubernetes.io/projected/09a7609b-8c62-48f7-a1f3-ec487d428a76-kube-api-access-2hqxq\") pod \"maas-controller-5f4c996689-xqs7f\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:51.329238 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.329202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f502cc28-13e7-462f-b79f-7d6840514ff1-webhook-cert\") pod \"maas-controller-864fc8cdd8-gjjfq\" (UID: \"f502cc28-13e7-462f-b79f-7d6840514ff1\") " pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:51.329438 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.329299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8lf\" (UniqueName: \"kubernetes.io/projected/f502cc28-13e7-462f-b79f-7d6840514ff1-kube-api-access-8q8lf\") pod \"maas-controller-864fc8cdd8-gjjfq\" (UID: \"f502cc28-13e7-462f-b79f-7d6840514ff1\") " pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:51.331844 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.331814 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f502cc28-13e7-462f-b79f-7d6840514ff1-webhook-cert\") pod \"maas-controller-864fc8cdd8-gjjfq\" (UID: \"f502cc28-13e7-462f-b79f-7d6840514ff1\") " pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:51.336847 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.336822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8lf\" (UniqueName: \"kubernetes.io/projected/f502cc28-13e7-462f-b79f-7d6840514ff1-kube-api-access-8q8lf\") pod \"maas-controller-864fc8cdd8-gjjfq\" (UID: \"f502cc28-13e7-462f-b79f-7d6840514ff1\") " pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:51.424063 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.424028 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:51.733868 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.733833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert\") pod \"maas-controller-5f4c996689-xqs7f\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:51.736267 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.736243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert\") pod \"maas-controller-5f4c996689-xqs7f\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:51.765653 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.765628 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-gjjfq"] Jul 30 20:28:51.767159 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:28:51.767132 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf502cc28_13e7_462f_b79f_7d6840514ff1.slice/crio-864fa1ad7d91440bc0e52bc7e14ad2c593d658ff3450e3498680e28986b1f523 WatchSource:0}: Error finding container 864fa1ad7d91440bc0e52bc7e14ad2c593d658ff3450e3498680e28986b1f523: Status 404 returned error can't find the container with id 864fa1ad7d91440bc0e52bc7e14ad2c593d658ff3450e3498680e28986b1f523 Jul 30 20:28:51.874849 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:51.874812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:52.013814 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:28:52.013660 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a7609b_8c62_48f7_a1f3_ec487d428a76.slice/crio-f986cbf3371c28f71dd63ae97bae7cf1f081cf197bfe9d483934e78b9a8f7f49 WatchSource:0}: Error finding container f986cbf3371c28f71dd63ae97bae7cf1f081cf197bfe9d483934e78b9a8f7f49: Status 404 returned error can't find the container with id f986cbf3371c28f71dd63ae97bae7cf1f081cf197bfe9d483934e78b9a8f7f49 Jul 30 20:28:52.022650 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:52.022626 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f4c996689-xqs7f"] Jul 30 20:28:52.388153 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:52.388038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" event={"ID":"f502cc28-13e7-462f-b79f-7d6840514ff1","Type":"ContainerStarted","Data":"864fa1ad7d91440bc0e52bc7e14ad2c593d658ff3450e3498680e28986b1f523"} Jul 30 20:28:52.389196 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:52.389173 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f4c996689-xqs7f" event={"ID":"09a7609b-8c62-48f7-a1f3-ec487d428a76","Type":"ContainerStarted","Data":"f986cbf3371c28f71dd63ae97bae7cf1f081cf197bfe9d483934e78b9a8f7f49"} Jul 30 20:28:55.408637 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:55.408600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f4c996689-xqs7f" event={"ID":"09a7609b-8c62-48f7-a1f3-ec487d428a76","Type":"ContainerStarted","Data":"9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e"} Jul 30 20:28:55.409100 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:55.408709 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:28:55.410195 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:55.410165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" event={"ID":"f502cc28-13e7-462f-b79f-7d6840514ff1","Type":"ContainerStarted","Data":"06140bfbfd7416eff23995689950e56ce3a3e2036abd091823d9d99686f7b55d"} Jul 30 20:28:55.410314 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:55.410296 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:28:55.428839 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:55.428791 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f4c996689-xqs7f" podStartSLOduration=3.045074609 podStartE2EDuration="5.428775458s" totalImagesPullingTime="2.383700849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:28:55.427550455 +0000 UTC m=+473.892688808" watchObservedRunningTime="2026-07-30 20:28:55.428775458 +0000 UTC m=+473.893913806" Jul 30 20:28:55.452325 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:28:55.452271 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" podStartSLOduration=1.8221614860000002 podStartE2EDuration="4.452256606s" totalImagesPullingTime="2.63009512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:28:55.45053691 +0000 UTC m=+473.915675260" watchObservedRunningTime="2026-07-30 20:28:55.452256606 +0000 UTC m=+473.917394954" Jul 30 20:29:06.417550 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.417510 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:29:06.417978 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.417594 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:29:06.458117 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.458080 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f4c996689-xqs7f"] Jul 30 20:29:06.468346 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.468303 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5f4c996689-xqs7f" podUID="09a7609b-8c62-48f7-a1f3-ec487d428a76" containerName="manager" containerID="cri-o://9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e" gracePeriod=30 Jul 30 20:29:06.715299 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.715273 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:29:06.756617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.756581 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert\") pod \"09a7609b-8c62-48f7-a1f3-ec487d428a76\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " Jul 30 20:29:06.756777 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.756741 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hqxq\" (UniqueName: \"kubernetes.io/projected/09a7609b-8c62-48f7-a1f3-ec487d428a76-kube-api-access-2hqxq\") pod \"09a7609b-8c62-48f7-a1f3-ec487d428a76\" (UID: \"09a7609b-8c62-48f7-a1f3-ec487d428a76\") " Jul 30 20:29:06.758860 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.758826 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a7609b-8c62-48f7-a1f3-ec487d428a76-kube-api-access-2hqxq" (OuterVolumeSpecName: "kube-api-access-2hqxq") pod "09a7609b-8c62-48f7-a1f3-ec487d428a76" (UID: "09a7609b-8c62-48f7-a1f3-ec487d428a76"). InnerVolumeSpecName "kube-api-access-2hqxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:29:06.758860 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.758832 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "09a7609b-8c62-48f7-a1f3-ec487d428a76" (UID: "09a7609b-8c62-48f7-a1f3-ec487d428a76"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:29:06.857524 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.857481 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09a7609b-8c62-48f7-a1f3-ec487d428a76-webhook-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:29:06.857524 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:06.857517 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hqxq\" (UniqueName: \"kubernetes.io/projected/09a7609b-8c62-48f7-a1f3-ec487d428a76-kube-api-access-2hqxq\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:29:07.474951 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.474912 2575 generic.go:356] "Generic (PLEG): container finished" podID="09a7609b-8c62-48f7-a1f3-ec487d428a76" containerID="9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e" exitCode=0 Jul 30 20:29:07.475387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.474957 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f4c996689-xqs7f" Jul 30 20:29:07.475387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.474997 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f4c996689-xqs7f" event={"ID":"09a7609b-8c62-48f7-a1f3-ec487d428a76","Type":"ContainerDied","Data":"9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e"} Jul 30 20:29:07.475387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.475077 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f4c996689-xqs7f" event={"ID":"09a7609b-8c62-48f7-a1f3-ec487d428a76","Type":"ContainerDied","Data":"f986cbf3371c28f71dd63ae97bae7cf1f081cf197bfe9d483934e78b9a8f7f49"} Jul 30 20:29:07.475387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.475108 2575 scope.go:117] "RemoveContainer" containerID="9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e" Jul 30 20:29:07.484491 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.484467 2575 scope.go:117] "RemoveContainer" containerID="9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e" Jul 30 20:29:07.484831 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:29:07.484799 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e\": container with ID starting with 9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e not found: ID does not exist" containerID="9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e" Jul 30 20:29:07.484927 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.484867 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e"} err="failed to get container status \"9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e\": rpc error: code = NotFound desc = could not find container \"9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e\": container with ID starting with 9c6287f55efdc69aadb7025a2af0d5a9c5078db3aabb537c478075bb668c3b7e not found: ID does not exist" Jul 30 20:29:07.498796 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.498761 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f4c996689-xqs7f"] Jul 30 20:29:07.500983 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:07.500948 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f4c996689-xqs7f"] Jul 30 20:29:08.012876 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:08.012843 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a7609b-8c62-48f7-a1f3-ec487d428a76" path="/var/lib/kubelet/pods/09a7609b-8c62-48f7-a1f3-ec487d428a76/volumes" Jul 30 20:29:27.744989 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.744948 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g"] Jul 30 20:29:27.745397 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.745306 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a7609b-8c62-48f7-a1f3-ec487d428a76" containerName="manager" Jul 30 20:29:27.745397 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.745318 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a7609b-8c62-48f7-a1f3-ec487d428a76" containerName="manager" Jul 30 20:29:27.745397 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.745386 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a7609b-8c62-48f7-a1f3-ec487d428a76" containerName="manager" Jul 30 20:29:27.752979 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.752946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:27.755668 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.755646 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 30 20:29:27.755864 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.755646 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vmvjr\"" Jul 30 20:29:27.760174 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.760145 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g"] Jul 30 20:29:27.781497 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.781458 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-69rsg"] Jul 30 20:29:27.784863 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.784840 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:27.795533 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.795504 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-69rsg"] Jul 30 20:29:27.839217 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.839184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-69rsg\" (UID: \"b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:27.839391 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.839232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jqp\" (UniqueName: \"kubernetes.io/projected/18c9c9df-de62-402b-9f53-acd7b1ed728a-kube-api-access-m2jqp\") pod \"payload-pre-processing-687dbb6cf6-zxl5g\" (UID: \"18c9c9df-de62-402b-9f53-acd7b1ed728a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:27.839391 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.839257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2fl\" (UniqueName: \"kubernetes.io/projected/b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8-kube-api-access-6v2fl\") pod \"payload-processing-5cc6cd444b-69rsg\" (UID: \"b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:27.839391 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.839322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c9c9df-de62-402b-9f53-acd7b1ed728a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zxl5g\" (UID: \"18c9c9df-de62-402b-9f53-acd7b1ed728a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:27.939851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.939814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c9c9df-de62-402b-9f53-acd7b1ed728a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zxl5g\" (UID: \"18c9c9df-de62-402b-9f53-acd7b1ed728a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:27.940049 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.939874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-69rsg\" (UID: \"b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:27.940049 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.939902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jqp\" (UniqueName: \"kubernetes.io/projected/18c9c9df-de62-402b-9f53-acd7b1ed728a-kube-api-access-m2jqp\") pod \"payload-pre-processing-687dbb6cf6-zxl5g\" (UID: \"18c9c9df-de62-402b-9f53-acd7b1ed728a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:27.940049 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.939918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2fl\" (UniqueName: \"kubernetes.io/projected/b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8-kube-api-access-6v2fl\") pod \"payload-processing-5cc6cd444b-69rsg\" (UID: \"b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:27.942038 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.941104 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-69rsg\" (UID: \"b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:27.942038 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.941104 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c9c9df-de62-402b-9f53-acd7b1ed728a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zxl5g\" (UID: \"18c9c9df-de62-402b-9f53-acd7b1ed728a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:27.948527 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.948492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2fl\" (UniqueName: \"kubernetes.io/projected/b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8-kube-api-access-6v2fl\") pod \"payload-processing-5cc6cd444b-69rsg\" (UID: \"b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:27.948840 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:27.948815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jqp\" (UniqueName: \"kubernetes.io/projected/18c9c9df-de62-402b-9f53-acd7b1ed728a-kube-api-access-m2jqp\") pod \"payload-pre-processing-687dbb6cf6-zxl5g\" (UID: \"18c9c9df-de62-402b-9f53-acd7b1ed728a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:28.066546 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:28.066459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:28.097594 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:28.097560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:28.224603 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:28.224567 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g"] Jul 30 20:29:28.227301 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:29:28.227271 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c9c9df_de62_402b_9f53_acd7b1ed728a.slice/crio-8536292f66a77dcb61da6a0730849265b7c8ef15782248443cfce904b7389611 WatchSource:0}: Error finding container 8536292f66a77dcb61da6a0730849265b7c8ef15782248443cfce904b7389611: Status 404 returned error can't find the container with id 8536292f66a77dcb61da6a0730849265b7c8ef15782248443cfce904b7389611 Jul 30 20:29:28.248702 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:28.248673 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-69rsg"] Jul 30 20:29:28.251716 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:29:28.251685 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb066fe25_58b5_4dd3_838a_2d8c0fa9b4f8.slice/crio-89500fb50adb490e871331f58d8f2c36f5cd4e7594df8b784c04cb1e8b82e7e0 WatchSource:0}: Error finding container 89500fb50adb490e871331f58d8f2c36f5cd4e7594df8b784c04cb1e8b82e7e0: Status 404 returned error can't find the container with id 89500fb50adb490e871331f58d8f2c36f5cd4e7594df8b784c04cb1e8b82e7e0 Jul 30 20:29:28.588766 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:28.588732 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" event={"ID":"b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8","Type":"ContainerStarted","Data":"89500fb50adb490e871331f58d8f2c36f5cd4e7594df8b784c04cb1e8b82e7e0"} Jul 30 20:29:28.589977 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:28.589947 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" event={"ID":"18c9c9df-de62-402b-9f53-acd7b1ed728a","Type":"ContainerStarted","Data":"8536292f66a77dcb61da6a0730849265b7c8ef15782248443cfce904b7389611"} Jul 30 20:29:30.604606 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:30.604562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" event={"ID":"18c9c9df-de62-402b-9f53-acd7b1ed728a","Type":"ContainerStarted","Data":"31b1ea352fd010e48dad2a57e0d52bb6721970ab82a38faa0bddea8b46a5dcc0"} Jul 30 20:29:30.605035 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:30.604704 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:30.606108 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:30.606088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" event={"ID":"b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8","Type":"ContainerStarted","Data":"b1f7436769a71c4ec866774a777e0ef0741d486a89ad1364fe1873ea205fb31c"} Jul 30 20:29:30.606229 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:30.606131 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:29:30.622116 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:30.622049 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" podStartSLOduration=1.715854415 podStartE2EDuration="3.622034555s" totalImagesPullingTime="1.90618014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:29:30.620747472 +0000 UTC m=+509.085885846" watchObservedRunningTime="2026-07-30 20:29:30.622034555 +0000 UTC m=+509.087172902" Jul 30 20:29:30.637754 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:30.637704 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" podStartSLOduration=1.7485458440000001 podStartE2EDuration="3.63768911s" totalImagesPullingTime="1.889143266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:29:30.636009794 +0000 UTC m=+509.101148145" watchObservedRunningTime="2026-07-30 20:29:30.63768911 +0000 UTC m=+509.102827459" Jul 30 20:29:41.612529 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:41.612491 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zxl5g" Jul 30 20:29:41.612934 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:29:41.612669 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-69rsg" Jul 30 20:30:00.140113 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:00.139996 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94"] Jul 30 20:30:00.143739 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:00.143716 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" Jul 30 20:30:00.146318 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:00.146288 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 20:30:00.151227 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:00.150886 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94"] Jul 30 20:30:00.154715 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:00.154697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" Jul 30 20:30:00.286289 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:00.286263 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94"] Jul 30 20:30:00.287886 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:30:00.287856 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5619676f_bc7a_4291_8050_4593f589263a.slice/crio-ba81a75d8812e163e1f8ef94edc086838bb1ed5507296eef403f0e4955f91f45 WatchSource:0}: Error finding container ba81a75d8812e163e1f8ef94edc086838bb1ed5507296eef403f0e4955f91f45: Status 404 returned error can't find the container with id ba81a75d8812e163e1f8ef94edc086838bb1ed5507296eef403f0e4955f91f45 Jul 30 20:30:00.766252 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:00.766216 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" event={"ID":"5619676f-bc7a-4291-8050-4593f589263a","Type":"ContainerStarted","Data":"ba81a75d8812e163e1f8ef94edc086838bb1ed5507296eef403f0e4955f91f45"} Jul 30 20:30:03.783879 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:03.783841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" event={"ID":"5619676f-bc7a-4291-8050-4593f589263a","Type":"ContainerStarted","Data":"5409286269ff6a269e9f020cc1a8fc8137f434f8031703a269ca6af32b6b518a"} Jul 30 20:30:03.799954 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:03.799902 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" podStartSLOduration=0.760598909 podStartE2EDuration="3.799888277s" totalImagesPullingTime="3.039289368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:30:03.798597721 +0000 UTC m=+542.263736067" watchObservedRunningTime="2026-07-30 20:30:03.799888277 +0000 UTC m=+542.265026623" Jul 30 20:30:04.789825 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:04.789785 2575 generic.go:356] "Generic (PLEG): container finished" podID="5619676f-bc7a-4291-8050-4593f589263a" containerID="5409286269ff6a269e9f020cc1a8fc8137f434f8031703a269ca6af32b6b518a" exitCode=0 Jul 30 20:30:04.790201 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:04.789840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" event={"ID":"5619676f-bc7a-4291-8050-4593f589263a","Type":"ContainerDied","Data":"5409286269ff6a269e9f020cc1a8fc8137f434f8031703a269ca6af32b6b518a"} Jul 30 20:30:05.925783 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:05.925757 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" Jul 30 20:30:06.801346 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:06.801320 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" Jul 30 20:30:06.801515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:06.801317 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94" event={"ID":"5619676f-bc7a-4291-8050-4593f589263a","Type":"ContainerDied","Data":"ba81a75d8812e163e1f8ef94edc086838bb1ed5507296eef403f0e4955f91f45"} Jul 30 20:30:06.801515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:06.801451 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba81a75d8812e163e1f8ef94edc086838bb1ed5507296eef403f0e4955f91f45" Jul 30 20:30:25.708986 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.708949 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt"] Jul 30 20:30:25.709453 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.709297 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5619676f-bc7a-4291-8050-4593f589263a" containerName="cleanup" Jul 30 20:30:25.709453 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.709309 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5619676f-bc7a-4291-8050-4593f589263a" containerName="cleanup" Jul 30 20:30:25.709453 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.709375 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5619676f-bc7a-4291-8050-4593f589263a" containerName="cleanup" Jul 30 20:30:25.712537 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.712520 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.716638 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.716605 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 30 20:30:25.716793 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.716613 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-67tzk\"" Jul 30 20:30:25.716793 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.716726 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 20:30:25.716893 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.716793 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 20:30:25.747965 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.747934 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt"] Jul 30 20:30:25.819228 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.819191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.819228 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.819230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.819478 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.819261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.819478 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.819287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.819478 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.819349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ldm\" (UniqueName: \"kubernetes.io/projected/d570968d-e30a-48d6-98ec-136bcfd9d030-kube-api-access-99ldm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.819478 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.819415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d570968d-e30a-48d6-98ec-136bcfd9d030-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.920546 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.920510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ldm\" (UniqueName: \"kubernetes.io/projected/d570968d-e30a-48d6-98ec-136bcfd9d030-kube-api-access-99ldm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.920999 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.920580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d570968d-e30a-48d6-98ec-136bcfd9d030-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.920999 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.920612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.920999 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.920629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.920999 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.920648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.920999 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.920676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.921408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.921082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.921408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.921182 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.921408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.921220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.923140 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.923112 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d570968d-e30a-48d6-98ec-136bcfd9d030-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.923506 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.923485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d570968d-e30a-48d6-98ec-136bcfd9d030-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:25.929224 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:25.929195 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ldm\" (UniqueName: \"kubernetes.io/projected/d570968d-e30a-48d6-98ec-136bcfd9d030-kube-api-access-99ldm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkgtt\" (UID: \"d570968d-e30a-48d6-98ec-136bcfd9d030\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:26.023694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:26.023603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:26.156178 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:26.156150 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt"] Jul 30 20:30:26.158208 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:30:26.158178 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd570968d_e30a_48d6_98ec_136bcfd9d030.slice/crio-a0160468f4cd362cfdb04958724cf1020f424756c7f832359a30ed885ed0642d WatchSource:0}: Error finding container a0160468f4cd362cfdb04958724cf1020f424756c7f832359a30ed885ed0642d: Status 404 returned error can't find the container with id a0160468f4cd362cfdb04958724cf1020f424756c7f832359a30ed885ed0642d Jul 30 20:30:26.903470 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:26.903430 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" event={"ID":"d570968d-e30a-48d6-98ec-136bcfd9d030","Type":"ContainerStarted","Data":"a0160468f4cd362cfdb04958724cf1020f424756c7f832359a30ed885ed0642d"} Jul 30 20:30:27.910275 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:27.910240 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" event={"ID":"d570968d-e30a-48d6-98ec-136bcfd9d030","Type":"ContainerStarted","Data":"05a0bdb39d2d2115e06da76b2f4eb5eed2017ffc3075014f7077553ff1eaae55"} Jul 30 20:30:27.929207 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:27.929152 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" podStartSLOduration=1.645492805 podStartE2EDuration="2.929135271s" totalImagesPullingTime="1.283642466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:30:27.926699381 +0000 UTC m=+566.391837742" watchObservedRunningTime="2026-07-30 20:30:27.929135271 +0000 UTC m=+566.394273619" Jul 30 20:30:30.360032 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.359989 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5"] Jul 30 20:30:30.363842 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.363816 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.366676 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.366652 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 30 20:30:30.373408 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.373371 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5"] Jul 30 20:30:30.465138 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.465086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cml5r\" (UniqueName: \"kubernetes.io/projected/0afc292b-b2bf-420b-9309-95c3e03f10e9-kube-api-access-cml5r\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.465353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.465149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.465353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.465202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0afc292b-b2bf-420b-9309-95c3e03f10e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.465353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.465270 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.465353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.465325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.465515 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.465352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566188 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566374 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566374 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566374 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cml5r\" (UniqueName: \"kubernetes.io/projected/0afc292b-b2bf-420b-9309-95c3e03f10e9-kube-api-access-cml5r\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566374 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566374 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0afc292b-b2bf-420b-9309-95c3e03f10e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566668 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566719 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.566759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.566720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.568599 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.568575 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0afc292b-b2bf-420b-9309-95c3e03f10e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.568784 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.568767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0afc292b-b2bf-420b-9309-95c3e03f10e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.574486 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.574457 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cml5r\" (UniqueName: \"kubernetes.io/projected/0afc292b-b2bf-420b-9309-95c3e03f10e9-kube-api-access-cml5r\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fhlj5\" (UID: \"0afc292b-b2bf-420b-9309-95c3e03f10e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.675174 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.675043 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:30.820923 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.820895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5"] Jul 30 20:30:30.823328 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:30:30.823288 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afc292b_b2bf_420b_9309_95c3e03f10e9.slice/crio-034b0437871c7ae2db2efc2f6131763dcb7bab1f18737581ce420e3756223974 WatchSource:0}: Error finding container 034b0437871c7ae2db2efc2f6131763dcb7bab1f18737581ce420e3756223974: Status 404 returned error can't find the container with id 034b0437871c7ae2db2efc2f6131763dcb7bab1f18737581ce420e3756223974 Jul 30 20:30:30.926474 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:30.926377 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" event={"ID":"0afc292b-b2bf-420b-9309-95c3e03f10e9","Type":"ContainerStarted","Data":"034b0437871c7ae2db2efc2f6131763dcb7bab1f18737581ce420e3756223974"} Jul 30 20:30:36.023773 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:36.023741 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:36.024280 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:36.023820 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:36.036810 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:36.036785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:36.959544 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:36.959503 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" event={"ID":"0afc292b-b2bf-420b-9309-95c3e03f10e9","Type":"ContainerStarted","Data":"e09052b580bf8504a726eff5465fd16384a2d8ae4cfc4bb2e06f175801060f27"} Jul 30 20:30:36.971450 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:36.971424 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkgtt" Jul 30 20:30:36.978670 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:36.978614 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" podStartSLOduration=1.632384886 podStartE2EDuration="6.978598305s" totalImagesPullingTime="5.346213419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:30:36.976549093 +0000 UTC m=+575.441687444" watchObservedRunningTime="2026-07-30 20:30:36.978598305 +0000 UTC m=+575.443736669" Jul 30 20:30:40.675952 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:40.675854 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:40.675952 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:40.675931 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:40.688352 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:40.688319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:30:40.990690 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:30:40.990653 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fhlj5" Jul 30 20:31:02.000545 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:31:02.000517 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:31:02.001808 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:31:02.001787 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:34:21.909398 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:21.909350 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-gjjfq"] Jul 30 20:34:21.910002 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:21.909625 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" podUID="f502cc28-13e7-462f-b79f-7d6840514ff1" containerName="manager" containerID="cri-o://06140bfbfd7416eff23995689950e56ce3a3e2036abd091823d9d99686f7b55d" gracePeriod=30 Jul 30 20:34:22.105400 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.105368 2575 generic.go:356] "Generic (PLEG): container finished" podID="f502cc28-13e7-462f-b79f-7d6840514ff1" containerID="06140bfbfd7416eff23995689950e56ce3a3e2036abd091823d9d99686f7b55d" exitCode=0 Jul 30 20:34:22.105546 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.105402 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" event={"ID":"f502cc28-13e7-462f-b79f-7d6840514ff1","Type":"ContainerDied","Data":"06140bfbfd7416eff23995689950e56ce3a3e2036abd091823d9d99686f7b55d"} Jul 30 20:34:22.159732 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.159672 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:34:22.256471 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.256434 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f502cc28-13e7-462f-b79f-7d6840514ff1-webhook-cert\") pod \"f502cc28-13e7-462f-b79f-7d6840514ff1\" (UID: \"f502cc28-13e7-462f-b79f-7d6840514ff1\") " Jul 30 20:34:22.256644 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.256511 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q8lf\" (UniqueName: \"kubernetes.io/projected/f502cc28-13e7-462f-b79f-7d6840514ff1-kube-api-access-8q8lf\") pod \"f502cc28-13e7-462f-b79f-7d6840514ff1\" (UID: \"f502cc28-13e7-462f-b79f-7d6840514ff1\") " Jul 30 20:34:22.258466 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.258440 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f502cc28-13e7-462f-b79f-7d6840514ff1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f502cc28-13e7-462f-b79f-7d6840514ff1" (UID: "f502cc28-13e7-462f-b79f-7d6840514ff1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:34:22.258558 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.258531 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f502cc28-13e7-462f-b79f-7d6840514ff1-kube-api-access-8q8lf" (OuterVolumeSpecName: "kube-api-access-8q8lf") pod "f502cc28-13e7-462f-b79f-7d6840514ff1" (UID: "f502cc28-13e7-462f-b79f-7d6840514ff1"). InnerVolumeSpecName "kube-api-access-8q8lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:34:22.358029 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.358000 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f502cc28-13e7-462f-b79f-7d6840514ff1-webhook-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:34:22.358029 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:22.358024 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q8lf\" (UniqueName: \"kubernetes.io/projected/f502cc28-13e7-462f-b79f-7d6840514ff1-kube-api-access-8q8lf\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:34:23.112151 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:23.112120 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" Jul 30 20:34:23.112151 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:23.112122 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-gjjfq" event={"ID":"f502cc28-13e7-462f-b79f-7d6840514ff1","Type":"ContainerDied","Data":"864fa1ad7d91440bc0e52bc7e14ad2c593d658ff3450e3498680e28986b1f523"} Jul 30 20:34:23.112636 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:23.112195 2575 scope.go:117] "RemoveContainer" containerID="06140bfbfd7416eff23995689950e56ce3a3e2036abd091823d9d99686f7b55d" Jul 30 20:34:23.137887 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:23.137854 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-gjjfq"] Jul 30 20:34:23.140351 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:23.140331 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-gjjfq"] Jul 30 20:34:24.014010 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:34:24.013967 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f502cc28-13e7-462f-b79f-7d6840514ff1" path="/var/lib/kubelet/pods/f502cc28-13e7-462f-b79f-7d6840514ff1/volumes" Jul 30 20:35:23.976685 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.976642 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-88pdd"] Jul 30 20:35:23.977205 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.977154 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f502cc28-13e7-462f-b79f-7d6840514ff1" containerName="manager" Jul 30 20:35:23.977205 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.977173 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f502cc28-13e7-462f-b79f-7d6840514ff1" containerName="manager" Jul 30 20:35:23.977327 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.977281 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f502cc28-13e7-462f-b79f-7d6840514ff1" containerName="manager" Jul 30 20:35:23.981458 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.981439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:23.983993 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.983971 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 20:35:23.985151 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.985129 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 20:35:23.985277 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.985135 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kgvj9\"" Jul 30 20:35:23.988276 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:23.988252 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-88pdd"] Jul 30 20:35:24.086964 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.086922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f329baf2-6cff-490e-8bdd-8570408ac4b3-webhook-cert\") pod \"maas-controller-864fc8cdd8-88pdd\" (UID: \"f329baf2-6cff-490e-8bdd-8570408ac4b3\") " pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:24.087190 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.086977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7x6g\" (UniqueName: \"kubernetes.io/projected/f329baf2-6cff-490e-8bdd-8570408ac4b3-kube-api-access-x7x6g\") pod \"maas-controller-864fc8cdd8-88pdd\" (UID: \"f329baf2-6cff-490e-8bdd-8570408ac4b3\") " pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:24.188461 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.188417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f329baf2-6cff-490e-8bdd-8570408ac4b3-webhook-cert\") pod \"maas-controller-864fc8cdd8-88pdd\" (UID: \"f329baf2-6cff-490e-8bdd-8570408ac4b3\") " pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:24.188615 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.188473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7x6g\" (UniqueName: \"kubernetes.io/projected/f329baf2-6cff-490e-8bdd-8570408ac4b3-kube-api-access-x7x6g\") pod \"maas-controller-864fc8cdd8-88pdd\" (UID: \"f329baf2-6cff-490e-8bdd-8570408ac4b3\") " pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:24.190870 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.190839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f329baf2-6cff-490e-8bdd-8570408ac4b3-webhook-cert\") pod \"maas-controller-864fc8cdd8-88pdd\" (UID: \"f329baf2-6cff-490e-8bdd-8570408ac4b3\") " pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:24.196960 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.196932 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7x6g\" (UniqueName: \"kubernetes.io/projected/f329baf2-6cff-490e-8bdd-8570408ac4b3-kube-api-access-x7x6g\") pod \"maas-controller-864fc8cdd8-88pdd\" (UID: \"f329baf2-6cff-490e-8bdd-8570408ac4b3\") " pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:24.294194 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.294110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:24.431575 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.431550 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-88pdd"] Jul 30 20:35:24.432863 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:35:24.432840 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf329baf2_6cff_490e_8bdd_8570408ac4b3.slice/crio-82af332d1494ae92962c48c9ada53d33d716f4b372d50141788f830812dd30ff WatchSource:0}: Error finding container 82af332d1494ae92962c48c9ada53d33d716f4b372d50141788f830812dd30ff: Status 404 returned error can't find the container with id 82af332d1494ae92962c48c9ada53d33d716f4b372d50141788f830812dd30ff Jul 30 20:35:24.434123 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:24.434104 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 20:35:25.431047 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:25.431006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" event={"ID":"f329baf2-6cff-490e-8bdd-8570408ac4b3","Type":"ContainerStarted","Data":"d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798"} Jul 30 20:35:25.431465 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:25.431098 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" event={"ID":"f329baf2-6cff-490e-8bdd-8570408ac4b3","Type":"ContainerStarted","Data":"82af332d1494ae92962c48c9ada53d33d716f4b372d50141788f830812dd30ff"} Jul 30 20:35:25.431465 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:25.431211 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:25.450434 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:25.450378 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" podStartSLOduration=1.943852243 podStartE2EDuration="2.450364317s" totalImagesPullingTime="506.512074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:35:25.447595075 +0000 UTC m=+863.912733424" watchObservedRunningTime="2026-07-30 20:35:25.450364317 +0000 UTC m=+863.915502663" Jul 30 20:35:36.438134 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:36.438104 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:41.092465 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.092429 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-88pdd"] Jul 30 20:35:41.092955 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.092686 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" podUID="f329baf2-6cff-490e-8bdd-8570408ac4b3" containerName="manager" containerID="cri-o://d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798" gracePeriod=30 Jul 30 20:35:41.346113 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.346037 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:41.425474 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.425443 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7x6g\" (UniqueName: \"kubernetes.io/projected/f329baf2-6cff-490e-8bdd-8570408ac4b3-kube-api-access-x7x6g\") pod \"f329baf2-6cff-490e-8bdd-8570408ac4b3\" (UID: \"f329baf2-6cff-490e-8bdd-8570408ac4b3\") " Jul 30 20:35:41.425634 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.425499 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f329baf2-6cff-490e-8bdd-8570408ac4b3-webhook-cert\") pod \"f329baf2-6cff-490e-8bdd-8570408ac4b3\" (UID: \"f329baf2-6cff-490e-8bdd-8570408ac4b3\") " Jul 30 20:35:41.427546 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.427509 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f329baf2-6cff-490e-8bdd-8570408ac4b3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f329baf2-6cff-490e-8bdd-8570408ac4b3" (UID: "f329baf2-6cff-490e-8bdd-8570408ac4b3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:35:41.427546 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.427536 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f329baf2-6cff-490e-8bdd-8570408ac4b3-kube-api-access-x7x6g" (OuterVolumeSpecName: "kube-api-access-x7x6g") pod "f329baf2-6cff-490e-8bdd-8570408ac4b3" (UID: "f329baf2-6cff-490e-8bdd-8570408ac4b3"). InnerVolumeSpecName "kube-api-access-x7x6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:35:41.517600 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.517559 2575 generic.go:356] "Generic (PLEG): container finished" podID="f329baf2-6cff-490e-8bdd-8570408ac4b3" containerID="d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798" exitCode=0 Jul 30 20:35:41.517777 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.517607 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" Jul 30 20:35:41.517777 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.517623 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" event={"ID":"f329baf2-6cff-490e-8bdd-8570408ac4b3","Type":"ContainerDied","Data":"d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798"} Jul 30 20:35:41.517777 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.517681 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-88pdd" event={"ID":"f329baf2-6cff-490e-8bdd-8570408ac4b3","Type":"ContainerDied","Data":"82af332d1494ae92962c48c9ada53d33d716f4b372d50141788f830812dd30ff"} Jul 30 20:35:41.517777 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.517713 2575 scope.go:117] "RemoveContainer" containerID="d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798" Jul 30 20:35:41.526406 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.526386 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7x6g\" (UniqueName: \"kubernetes.io/projected/f329baf2-6cff-490e-8bdd-8570408ac4b3-kube-api-access-x7x6g\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:35:41.526510 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.526411 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f329baf2-6cff-490e-8bdd-8570408ac4b3-webhook-cert\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:35:41.527245 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.527213 2575 scope.go:117] "RemoveContainer" containerID="d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798" Jul 30 20:35:41.527466 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:35:41.527450 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798\": container with ID starting with d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798 not found: ID does not exist" containerID="d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798" Jul 30 20:35:41.527516 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.527473 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798"} err="failed to get container status \"d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798\": rpc error: code = NotFound desc = could not find container \"d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798\": container with ID starting with d7d1afbced89192ddeeae8c0de7fc4615977be4ee5288e32d3302a50781bb798 not found: ID does not exist" Jul 30 20:35:41.540525 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.540499 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-88pdd"] Jul 30 20:35:41.543815 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:41.543793 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-88pdd"] Jul 30 20:35:42.013537 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.013505 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f329baf2-6cff-490e-8bdd-8570408ac4b3" path="/var/lib/kubelet/pods/f329baf2-6cff-490e-8bdd-8570408ac4b3/volumes" Jul 30 20:35:42.387101 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.387019 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-xvkkf"] Jul 30 20:35:42.387460 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.387382 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f329baf2-6cff-490e-8bdd-8570408ac4b3" containerName="manager" Jul 30 20:35:42.387460 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.387394 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f329baf2-6cff-490e-8bdd-8570408ac4b3" containerName="manager" Jul 30 20:35:42.387553 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.387474 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f329baf2-6cff-490e-8bdd-8570408ac4b3" containerName="manager" Jul 30 20:35:42.391756 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.391738 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:42.394288 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.394269 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 20:35:42.395287 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.395268 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 20:35:42.395380 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.395314 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kgvj9\"" Jul 30 20:35:42.400272 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.400254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-xvkkf"] Jul 30 20:35:42.433963 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.433940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/afce1a47-23af-44d9-b6ea-42aa7c60439f-webhook-cert\") pod \"maas-controller-864fc8cdd8-xvkkf\" (UID: \"afce1a47-23af-44d9-b6ea-42aa7c60439f\") " pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:42.434089 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.433991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgx8l\" (UniqueName: \"kubernetes.io/projected/afce1a47-23af-44d9-b6ea-42aa7c60439f-kube-api-access-pgx8l\") pod \"maas-controller-864fc8cdd8-xvkkf\" (UID: \"afce1a47-23af-44d9-b6ea-42aa7c60439f\") " pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:42.534748 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.534719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/afce1a47-23af-44d9-b6ea-42aa7c60439f-webhook-cert\") pod \"maas-controller-864fc8cdd8-xvkkf\" (UID: \"afce1a47-23af-44d9-b6ea-42aa7c60439f\") " pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:42.534883 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.534770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgx8l\" (UniqueName: \"kubernetes.io/projected/afce1a47-23af-44d9-b6ea-42aa7c60439f-kube-api-access-pgx8l\") pod \"maas-controller-864fc8cdd8-xvkkf\" (UID: \"afce1a47-23af-44d9-b6ea-42aa7c60439f\") " pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:42.537278 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.537255 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/afce1a47-23af-44d9-b6ea-42aa7c60439f-webhook-cert\") pod \"maas-controller-864fc8cdd8-xvkkf\" (UID: \"afce1a47-23af-44d9-b6ea-42aa7c60439f\") " pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:42.542817 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.542794 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgx8l\" (UniqueName: \"kubernetes.io/projected/afce1a47-23af-44d9-b6ea-42aa7c60439f-kube-api-access-pgx8l\") pod \"maas-controller-864fc8cdd8-xvkkf\" (UID: \"afce1a47-23af-44d9-b6ea-42aa7c60439f\") " pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:42.702897 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.702871 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:42.834759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:42.834733 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-864fc8cdd8-xvkkf"] Jul 30 20:35:42.836659 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:35:42.836634 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafce1a47_23af_44d9_b6ea_42aa7c60439f.slice/crio-585834521cf92356069b0eabf676c3ddce348469d6108029105d12d0b1510564 WatchSource:0}: Error finding container 585834521cf92356069b0eabf676c3ddce348469d6108029105d12d0b1510564: Status 404 returned error can't find the container with id 585834521cf92356069b0eabf676c3ddce348469d6108029105d12d0b1510564 Jul 30 20:35:43.529838 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:43.529801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" event={"ID":"afce1a47-23af-44d9-b6ea-42aa7c60439f","Type":"ContainerStarted","Data":"e6f724ca876f8da5beac246c38a651e24931d52df89b163ae655098dd92bb377"} Jul 30 20:35:43.530301 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:43.529876 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:35:43.530301 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:43.529898 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" event={"ID":"afce1a47-23af-44d9-b6ea-42aa7c60439f","Type":"ContainerStarted","Data":"585834521cf92356069b0eabf676c3ddce348469d6108029105d12d0b1510564"} Jul 30 20:35:43.549628 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:43.549575 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" podStartSLOduration=0.987380041 podStartE2EDuration="1.549559402s" totalImagesPullingTime="562.179361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:35:43.54670237 +0000 UTC m=+882.011840721" watchObservedRunningTime="2026-07-30 20:35:43.549559402 +0000 UTC m=+882.014697749" Jul 30 20:35:54.536950 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:35:54.536863 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-864fc8cdd8-xvkkf" Jul 30 20:36:02.038925 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:36:02.038894 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:36:02.039381 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:36:02.039352 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:41:02.071594 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:41:02.071519 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:41:02.073450 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:41:02.073427 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:46:02.107505 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:46:02.107479 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:46:02.110907 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:46:02.110885 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:50:51.054316 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.054268 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4"] Jul 30 20:50:51.057851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.057831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:51.060687 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.060666 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ae95a8d6-dockercfg-cg484\"" Jul 30 20:50:51.060808 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.060708 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ae95a8d6\"" Jul 30 20:50:51.069089 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.069037 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4"] Jul 30 20:50:51.096245 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.096209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6071f50-efd2-4aa5-b5c3-1718edefa218-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4\" (UID: \"e6071f50-efd2-4aa5-b5c3-1718edefa218\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:51.096439 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.096323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94xq\" (UniqueName: \"kubernetes.io/projected/e6071f50-efd2-4aa5-b5c3-1718edefa218-kube-api-access-n94xq\") pod \"payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4\" (UID: \"e6071f50-efd2-4aa5-b5c3-1718edefa218\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:51.133150 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.133105 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj"] Jul 30 20:50:51.136627 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.136601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:51.148355 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.148318 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj"] Jul 30 20:50:51.197441 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.197401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xr8\" (UniqueName: \"kubernetes.io/projected/13596902-8765-4b20-938f-550be172a272-kube-api-access-s6xr8\") pod \"payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj\" (UID: \"13596902-8765-4b20-938f-550be172a272\") " pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:51.197637 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.197452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6071f50-efd2-4aa5-b5c3-1718edefa218-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4\" (UID: \"e6071f50-efd2-4aa5-b5c3-1718edefa218\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:51.197637 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.197500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13596902-8765-4b20-938f-550be172a272-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj\" (UID: \"13596902-8765-4b20-938f-550be172a272\") " pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:51.197720 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.197668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94xq\" (UniqueName: \"kubernetes.io/projected/e6071f50-efd2-4aa5-b5c3-1718edefa218-kube-api-access-n94xq\") pod \"payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4\" (UID: \"e6071f50-efd2-4aa5-b5c3-1718edefa218\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:51.198288 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.198269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6071f50-efd2-4aa5-b5c3-1718edefa218-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4\" (UID: \"e6071f50-efd2-4aa5-b5c3-1718edefa218\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:51.206776 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.206746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94xq\" (UniqueName: \"kubernetes.io/projected/e6071f50-efd2-4aa5-b5c3-1718edefa218-kube-api-access-n94xq\") pod \"payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4\" (UID: \"e6071f50-efd2-4aa5-b5c3-1718edefa218\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:51.298799 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.298759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xr8\" (UniqueName: \"kubernetes.io/projected/13596902-8765-4b20-938f-550be172a272-kube-api-access-s6xr8\") pod \"payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj\" (UID: \"13596902-8765-4b20-938f-550be172a272\") " pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:51.298983 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.298808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13596902-8765-4b20-938f-550be172a272-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj\" (UID: \"13596902-8765-4b20-938f-550be172a272\") " pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:51.299640 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.299619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13596902-8765-4b20-938f-550be172a272-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj\" (UID: \"13596902-8765-4b20-938f-550be172a272\") " pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:51.308030 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.307952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xr8\" (UniqueName: \"kubernetes.io/projected/13596902-8765-4b20-938f-550be172a272-kube-api-access-s6xr8\") pod \"payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj\" (UID: \"13596902-8765-4b20-938f-550be172a272\") " pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:51.370112 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.370042 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:51.449082 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.449008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:51.527618 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.527583 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4"] Jul 30 20:50:51.529484 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:50:51.529446 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6071f50_efd2_4aa5_b5c3_1718edefa218.slice/crio-f953226bfd8aea139eb3a2a12dc1ba0ef109435105ac819fa4569fdae317e961 WatchSource:0}: Error finding container f953226bfd8aea139eb3a2a12dc1ba0ef109435105ac819fa4569fdae317e961: Status 404 returned error can't find the container with id f953226bfd8aea139eb3a2a12dc1ba0ef109435105ac819fa4569fdae317e961 Jul 30 20:50:51.615158 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:51.615119 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj"] Jul 30 20:50:51.616858 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:50:51.616831 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13596902_8765_4b20_938f_550be172a272.slice/crio-cd7bd8f6b55f9dfd9f2d3b7911efe19e41156c78b6076585746e772fd471e4f5 WatchSource:0}: Error finding container cd7bd8f6b55f9dfd9f2d3b7911efe19e41156c78b6076585746e772fd471e4f5: Status 404 returned error can't find the container with id cd7bd8f6b55f9dfd9f2d3b7911efe19e41156c78b6076585746e772fd471e4f5 Jul 30 20:50:52.297810 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:52.297762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" event={"ID":"13596902-8765-4b20-938f-550be172a272","Type":"ContainerStarted","Data":"083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115"} Jul 30 20:50:52.298324 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:52.297950 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:50:52.298324 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:52.297987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" event={"ID":"13596902-8765-4b20-938f-550be172a272","Type":"ContainerStarted","Data":"cd7bd8f6b55f9dfd9f2d3b7911efe19e41156c78b6076585746e772fd471e4f5"} Jul 30 20:50:52.299434 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:52.299413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" event={"ID":"e6071f50-efd2-4aa5-b5c3-1718edefa218","Type":"ContainerStarted","Data":"96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d"} Jul 30 20:50:52.299503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:52.299455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" event={"ID":"e6071f50-efd2-4aa5-b5c3-1718edefa218","Type":"ContainerStarted","Data":"f953226bfd8aea139eb3a2a12dc1ba0ef109435105ac819fa4569fdae317e961"} Jul 30 20:50:52.299557 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:52.299540 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:50:52.317507 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:52.317453 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" podStartSLOduration=1.317433495 podStartE2EDuration="1.317433495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:50:52.315565114 +0000 UTC m=+1790.780703464" watchObservedRunningTime="2026-07-30 20:50:52.317433495 +0000 UTC m=+1790.782571840" Jul 30 20:50:52.336234 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:50:52.336176 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" podStartSLOduration=1.336155317 podStartE2EDuration="1.336155317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:50:52.334291348 +0000 UTC m=+1790.799429697" watchObservedRunningTime="2026-07-30 20:50:52.336155317 +0000 UTC m=+1790.801293667" Jul 30 20:51:02.142862 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:02.142833 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:51:02.146519 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:02.146498 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:51:03.307011 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:03.306984 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:51:03.307446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:03.307068 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:51:14.719549 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:14.719512 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql"] Jul 30 20:51:14.722984 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:14.722961 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" Jul 30 20:51:14.726190 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:14.726165 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 20:51:14.734274 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:14.734250 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql"] Jul 30 20:51:14.812994 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:14.812952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce603220-3b38-4f4c-9ced-b702d61eb4ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql\" (UID: \"ce603220-3b38-4f4c-9ced-b702d61eb4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" Jul 30 20:51:14.914383 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:14.914340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce603220-3b38-4f4c-9ced-b702d61eb4ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql\" (UID: \"ce603220-3b38-4f4c-9ced-b702d61eb4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" Jul 30 20:51:14.914968 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:14.914947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce603220-3b38-4f4c-9ced-b702d61eb4ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql\" (UID: \"ce603220-3b38-4f4c-9ced-b702d61eb4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" Jul 30 20:51:15.035304 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:15.035212 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" Jul 30 20:51:15.180122 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:15.180090 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql"] Jul 30 20:51:15.182114 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:51:15.182079 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce603220_3b38_4f4c_9ced_b702d61eb4ee.slice/crio-89cb69733aced65ab12762195aac1c1c2ce2dce4faf507fc15e145abce501ad7 WatchSource:0}: Error finding container 89cb69733aced65ab12762195aac1c1c2ce2dce4faf507fc15e145abce501ad7: Status 404 returned error can't find the container with id 89cb69733aced65ab12762195aac1c1c2ce2dce4faf507fc15e145abce501ad7 Jul 30 20:51:15.427157 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:15.427076 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" event={"ID":"ce603220-3b38-4f4c-9ced-b702d61eb4ee","Type":"ContainerStarted","Data":"89cb69733aced65ab12762195aac1c1c2ce2dce4faf507fc15e145abce501ad7"} Jul 30 20:51:16.433790 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:16.433686 2575 generic.go:356] "Generic (PLEG): container finished" podID="ce603220-3b38-4f4c-9ced-b702d61eb4ee" containerID="6519ca7b00709d18da5450d6673ca0d6466bb8bbcc248b9d4147a309b54e4474" exitCode=0 Jul 30 20:51:16.433790 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:16.433776 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" event={"ID":"ce603220-3b38-4f4c-9ced-b702d61eb4ee","Type":"ContainerDied","Data":"6519ca7b00709d18da5450d6673ca0d6466bb8bbcc248b9d4147a309b54e4474"} Jul 30 20:51:17.583025 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:17.582994 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" Jul 30 20:51:17.737774 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:17.737715 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce603220-3b38-4f4c-9ced-b702d61eb4ee-maas-api-service-ca\") pod \"ce603220-3b38-4f4c-9ced-b702d61eb4ee\" (UID: \"ce603220-3b38-4f4c-9ced-b702d61eb4ee\") " Jul 30 20:51:17.738140 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:17.738116 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce603220-3b38-4f4c-9ced-b702d61eb4ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce603220-3b38-4f4c-9ced-b702d61eb4ee" (UID: "ce603220-3b38-4f4c-9ced-b702d61eb4ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:51:17.838781 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:17.838737 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce603220-3b38-4f4c-9ced-b702d61eb4ee-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:18.444963 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:18.444879 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" event={"ID":"ce603220-3b38-4f4c-9ced-b702d61eb4ee","Type":"ContainerDied","Data":"89cb69733aced65ab12762195aac1c1c2ce2dce4faf507fc15e145abce501ad7"} Jul 30 20:51:18.444963 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:18.444933 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89cb69733aced65ab12762195aac1c1c2ce2dce4faf507fc15e145abce501ad7" Jul 30 20:51:18.444963 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:18.444888 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql" Jul 30 20:51:20.884753 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:20.884717 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj"] Jul 30 20:51:20.885262 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:20.884983 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" podUID="13596902-8765-4b20-938f-550be172a272" containerName="payload-processing" containerID="cri-o://083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115" gracePeriod=30 Jul 30 20:51:20.892491 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:20.892101 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4"] Jul 30 20:51:20.892491 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:20.892327 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" podUID="e6071f50-efd2-4aa5-b5c3-1718edefa218" containerName="payload-pre-processing" containerID="cri-o://96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d" gracePeriod=30 Jul 30 20:51:21.027783 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.027742 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql"] Jul 30 20:51:21.030089 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.030044 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae95a8d6-dkkql"] Jul 30 20:51:21.144645 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.144571 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:51:21.184069 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.184032 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:51:21.271404 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.271366 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6xr8\" (UniqueName: \"kubernetes.io/projected/13596902-8765-4b20-938f-550be172a272-kube-api-access-s6xr8\") pod \"13596902-8765-4b20-938f-550be172a272\" (UID: \"13596902-8765-4b20-938f-550be172a272\") " Jul 30 20:51:21.271568 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.271420 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13596902-8765-4b20-938f-550be172a272-plugins-config-volume\") pod \"13596902-8765-4b20-938f-550be172a272\" (UID: \"13596902-8765-4b20-938f-550be172a272\") " Jul 30 20:51:21.271568 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.271458 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6071f50-efd2-4aa5-b5c3-1718edefa218-plugins-config-volume\") pod \"e6071f50-efd2-4aa5-b5c3-1718edefa218\" (UID: \"e6071f50-efd2-4aa5-b5c3-1718edefa218\") " Jul 30 20:51:21.271568 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.271508 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n94xq\" (UniqueName: \"kubernetes.io/projected/e6071f50-efd2-4aa5-b5c3-1718edefa218-kube-api-access-n94xq\") pod \"e6071f50-efd2-4aa5-b5c3-1718edefa218\" (UID: \"e6071f50-efd2-4aa5-b5c3-1718edefa218\") " Jul 30 20:51:21.271948 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.271916 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6071f50-efd2-4aa5-b5c3-1718edefa218-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6071f50-efd2-4aa5-b5c3-1718edefa218" (UID: "e6071f50-efd2-4aa5-b5c3-1718edefa218"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:51:21.272079 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.271955 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13596902-8765-4b20-938f-550be172a272-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13596902-8765-4b20-938f-550be172a272" (UID: "13596902-8765-4b20-938f-550be172a272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:51:21.273740 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.273714 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13596902-8765-4b20-938f-550be172a272-kube-api-access-s6xr8" (OuterVolumeSpecName: "kube-api-access-s6xr8") pod "13596902-8765-4b20-938f-550be172a272" (UID: "13596902-8765-4b20-938f-550be172a272"). InnerVolumeSpecName "kube-api-access-s6xr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:51:21.273740 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.273719 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6071f50-efd2-4aa5-b5c3-1718edefa218-kube-api-access-n94xq" (OuterVolumeSpecName: "kube-api-access-n94xq") pod "e6071f50-efd2-4aa5-b5c3-1718edefa218" (UID: "e6071f50-efd2-4aa5-b5c3-1718edefa218"). InnerVolumeSpecName "kube-api-access-n94xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:51:21.372304 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.372265 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xr8\" (UniqueName: \"kubernetes.io/projected/13596902-8765-4b20-938f-550be172a272-kube-api-access-s6xr8\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:21.372304 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.372296 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13596902-8765-4b20-938f-550be172a272-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:21.372304 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.372306 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6071f50-efd2-4aa5-b5c3-1718edefa218-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:21.372304 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.372314 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n94xq\" (UniqueName: \"kubernetes.io/projected/e6071f50-efd2-4aa5-b5c3-1718edefa218-kube-api-access-n94xq\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:21.462566 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.462529 2575 generic.go:356] "Generic (PLEG): container finished" podID="13596902-8765-4b20-938f-550be172a272" containerID="083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115" exitCode=0 Jul 30 20:51:21.462767 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.462597 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" Jul 30 20:51:21.462767 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.462624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" event={"ID":"13596902-8765-4b20-938f-550be172a272","Type":"ContainerDied","Data":"083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115"} Jul 30 20:51:21.462767 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.462699 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj" event={"ID":"13596902-8765-4b20-938f-550be172a272","Type":"ContainerDied","Data":"cd7bd8f6b55f9dfd9f2d3b7911efe19e41156c78b6076585746e772fd471e4f5"} Jul 30 20:51:21.462767 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.462726 2575 scope.go:117] "RemoveContainer" containerID="083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115" Jul 30 20:51:21.463993 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.463975 2575 generic.go:356] "Generic (PLEG): container finished" podID="e6071f50-efd2-4aa5-b5c3-1718edefa218" containerID="96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d" exitCode=0 Jul 30 20:51:21.464106 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.464073 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" Jul 30 20:51:21.464106 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.464083 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" event={"ID":"e6071f50-efd2-4aa5-b5c3-1718edefa218","Type":"ContainerDied","Data":"96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d"} Jul 30 20:51:21.464216 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.464143 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4" event={"ID":"e6071f50-efd2-4aa5-b5c3-1718edefa218","Type":"ContainerDied","Data":"f953226bfd8aea139eb3a2a12dc1ba0ef109435105ac819fa4569fdae317e961"} Jul 30 20:51:21.473469 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.473452 2575 scope.go:117] "RemoveContainer" containerID="083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115" Jul 30 20:51:21.473765 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:51:21.473734 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115\": container with ID starting with 083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115 not found: ID does not exist" containerID="083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115" Jul 30 20:51:21.473826 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.473776 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115"} err="failed to get container status \"083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115\": rpc error: code = NotFound desc = could not find container \"083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115\": container with ID starting with 083c0f77c479a51dafa761881abeff7fa1d751117084e18af41600b1ab447115 not found: ID does not exist" Jul 30 20:51:21.473826 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.473794 2575 scope.go:117] "RemoveContainer" containerID="96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d" Jul 30 20:51:21.482433 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.482410 2575 scope.go:117] "RemoveContainer" containerID="96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d" Jul 30 20:51:21.482693 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:51:21.482672 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d\": container with ID starting with 96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d not found: ID does not exist" containerID="96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d" Jul 30 20:51:21.482751 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.482703 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d"} err="failed to get container status \"96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d\": rpc error: code = NotFound desc = could not find container \"96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d\": container with ID starting with 96c80a47cc58ce34db1dc40764bb66595c3f936a4dce24e0974e77dcb7607c7d not found: ID does not exist" Jul 30 20:51:21.512419 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.512387 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj"] Jul 30 20:51:21.522657 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.522629 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae95a8d6-b489f4fc8-rrclj"] Jul 30 20:51:21.548469 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.548434 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4"] Jul 30 20:51:21.557024 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:21.556994 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae95a8d6-54fc78cb76-kmlc4"] Jul 30 20:51:22.014939 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:22.014904 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13596902-8765-4b20-938f-550be172a272" path="/var/lib/kubelet/pods/13596902-8765-4b20-938f-550be172a272/volumes" Jul 30 20:51:22.015397 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:22.015316 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce603220-3b38-4f4c-9ced-b702d61eb4ee" path="/var/lib/kubelet/pods/ce603220-3b38-4f4c-9ced-b702d61eb4ee/volumes" Jul 30 20:51:22.015640 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:22.015626 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6071f50-efd2-4aa5-b5c3-1718edefa218" path="/var/lib/kubelet/pods/e6071f50-efd2-4aa5-b5c3-1718edefa218/volumes" Jul 30 20:51:25.319527 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319490 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws"] Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319882 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13596902-8765-4b20-938f-550be172a272" containerName="payload-processing" Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319894 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13596902-8765-4b20-938f-550be172a272" containerName="payload-processing" Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319907 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce603220-3b38-4f4c-9ced-b702d61eb4ee" containerName="revoke-keys" Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319913 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce603220-3b38-4f4c-9ced-b702d61eb4ee" containerName="revoke-keys" Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319923 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6071f50-efd2-4aa5-b5c3-1718edefa218" containerName="payload-pre-processing" Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319928 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6071f50-efd2-4aa5-b5c3-1718edefa218" containerName="payload-pre-processing" Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319985 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce603220-3b38-4f4c-9ced-b702d61eb4ee" containerName="revoke-keys" Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.319993 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13596902-8765-4b20-938f-550be172a272" containerName="payload-processing" Jul 30 20:51:25.320018 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.320000 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6071f50-efd2-4aa5-b5c3-1718edefa218" containerName="payload-pre-processing" Jul 30 20:51:25.324575 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.324547 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts"] Jul 30 20:51:25.324714 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.324689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:25.327809 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.327786 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1f4c9be7\"" Jul 30 20:51:25.328033 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.328017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:25.331136 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.331112 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1f4c9be7-dockercfg-9826s\"" Jul 30 20:51:25.335759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.335732 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts"] Jul 30 20:51:25.342545 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.342261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws"] Jul 30 20:51:25.412551 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.412515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwkc\" (UniqueName: \"kubernetes.io/projected/3b82588d-56c8-44d9-b619-e2cd811285bc-kube-api-access-7xwkc\") pod \"payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws\" (UID: \"3b82588d-56c8-44d9-b619-e2cd811285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:25.412741 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.412567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49230553-6792-4f96-a086-79bcc9ccfcd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts\" (UID: \"49230553-6792-4f96-a086-79bcc9ccfcd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:25.412741 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.412628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tt6\" (UniqueName: \"kubernetes.io/projected/49230553-6792-4f96-a086-79bcc9ccfcd2-kube-api-access-t9tt6\") pod \"payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts\" (UID: \"49230553-6792-4f96-a086-79bcc9ccfcd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:25.412741 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.412698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b82588d-56c8-44d9-b619-e2cd811285bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws\" (UID: \"3b82588d-56c8-44d9-b619-e2cd811285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:25.513729 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.513686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b82588d-56c8-44d9-b619-e2cd811285bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws\" (UID: \"3b82588d-56c8-44d9-b619-e2cd811285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:25.513944 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.513742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwkc\" (UniqueName: \"kubernetes.io/projected/3b82588d-56c8-44d9-b619-e2cd811285bc-kube-api-access-7xwkc\") pod \"payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws\" (UID: \"3b82588d-56c8-44d9-b619-e2cd811285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:25.513944 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.513778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49230553-6792-4f96-a086-79bcc9ccfcd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts\" (UID: \"49230553-6792-4f96-a086-79bcc9ccfcd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:25.513944 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.513811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tt6\" (UniqueName: \"kubernetes.io/projected/49230553-6792-4f96-a086-79bcc9ccfcd2-kube-api-access-t9tt6\") pod \"payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts\" (UID: \"49230553-6792-4f96-a086-79bcc9ccfcd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:25.514591 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.514566 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b82588d-56c8-44d9-b619-e2cd811285bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws\" (UID: \"3b82588d-56c8-44d9-b619-e2cd811285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:25.514694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.514566 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49230553-6792-4f96-a086-79bcc9ccfcd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts\" (UID: \"49230553-6792-4f96-a086-79bcc9ccfcd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:25.525929 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.525897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwkc\" (UniqueName: \"kubernetes.io/projected/3b82588d-56c8-44d9-b619-e2cd811285bc-kube-api-access-7xwkc\") pod \"payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws\" (UID: \"3b82588d-56c8-44d9-b619-e2cd811285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:25.525929 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.525900 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tt6\" (UniqueName: \"kubernetes.io/projected/49230553-6792-4f96-a086-79bcc9ccfcd2-kube-api-access-t9tt6\") pod \"payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts\" (UID: \"49230553-6792-4f96-a086-79bcc9ccfcd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:25.637131 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.637024 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:25.646469 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.646444 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:25.798424 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.798327 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws"] Jul 30 20:51:25.800589 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:51:25.800558 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b82588d_56c8_44d9_b619_e2cd811285bc.slice/crio-7b28941f61afd0a52915e71c83f4b26a39f0cfaab457ec2912bdc2ef6c8e02f7 WatchSource:0}: Error finding container 7b28941f61afd0a52915e71c83f4b26a39f0cfaab457ec2912bdc2ef6c8e02f7: Status 404 returned error can't find the container with id 7b28941f61afd0a52915e71c83f4b26a39f0cfaab457ec2912bdc2ef6c8e02f7 Jul 30 20:51:25.827487 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:25.827464 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts"] Jul 30 20:51:25.833698 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:51:25.833671 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49230553_6792_4f96_a086_79bcc9ccfcd2.slice/crio-eeea26de989e610fbbd89e0a9202f2da8aca07dacab75b4044d1b43929c1cf99 WatchSource:0}: Error finding container eeea26de989e610fbbd89e0a9202f2da8aca07dacab75b4044d1b43929c1cf99: Status 404 returned error can't find the container with id eeea26de989e610fbbd89e0a9202f2da8aca07dacab75b4044d1b43929c1cf99 Jul 30 20:51:26.495835 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:26.495790 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" event={"ID":"3b82588d-56c8-44d9-b619-e2cd811285bc","Type":"ContainerStarted","Data":"cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e"} Jul 30 20:51:26.496267 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:26.495853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" event={"ID":"3b82588d-56c8-44d9-b619-e2cd811285bc","Type":"ContainerStarted","Data":"7b28941f61afd0a52915e71c83f4b26a39f0cfaab457ec2912bdc2ef6c8e02f7"} Jul 30 20:51:26.496267 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:26.495994 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:26.497492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:26.497464 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" event={"ID":"49230553-6792-4f96-a086-79bcc9ccfcd2","Type":"ContainerStarted","Data":"9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318"} Jul 30 20:51:26.497595 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:26.497515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" event={"ID":"49230553-6792-4f96-a086-79bcc9ccfcd2","Type":"ContainerStarted","Data":"eeea26de989e610fbbd89e0a9202f2da8aca07dacab75b4044d1b43929c1cf99"} Jul 30 20:51:26.497595 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:26.497554 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:26.528522 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:26.528459 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" podStartSLOduration=1.528437651 podStartE2EDuration="1.528437651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:51:26.525355537 +0000 UTC m=+1824.990493885" watchObservedRunningTime="2026-07-30 20:51:26.528437651 +0000 UTC m=+1824.993576000" Jul 30 20:51:26.550853 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:26.550797 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" podStartSLOduration=1.5507762029999999 podStartE2EDuration="1.550776203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:51:26.54793596 +0000 UTC m=+1825.013074310" watchObservedRunningTime="2026-07-30 20:51:26.550776203 +0000 UTC m=+1825.015914551" Jul 30 20:51:37.504858 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:37.504826 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:37.505255 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:37.504900 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:49.437811 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.437771 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q"] Jul 30 20:51:49.441280 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.441260 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" Jul 30 20:51:49.450557 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.450532 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 20:51:49.459497 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.459467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q"] Jul 30 20:51:49.532254 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.532210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e69e04-a20b-4e2f-a9b3-3f2b48239d20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q\" (UID: \"41e69e04-a20b-4e2f-a9b3-3f2b48239d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" Jul 30 20:51:49.633640 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.633608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e69e04-a20b-4e2f-a9b3-3f2b48239d20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q\" (UID: \"41e69e04-a20b-4e2f-a9b3-3f2b48239d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" Jul 30 20:51:49.634289 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.634268 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e69e04-a20b-4e2f-a9b3-3f2b48239d20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q\" (UID: \"41e69e04-a20b-4e2f-a9b3-3f2b48239d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" Jul 30 20:51:49.752671 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.752636 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" Jul 30 20:51:49.903993 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:49.903965 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q"] Jul 30 20:51:49.905703 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:51:49.905664 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e69e04_a20b_4e2f_a9b3_3f2b48239d20.slice/crio-d85db9378072c88b101bb76b647e9732dc1c363a1ce2a2d261400f19914e9936 WatchSource:0}: Error finding container d85db9378072c88b101bb76b647e9732dc1c363a1ce2a2d261400f19914e9936: Status 404 returned error can't find the container with id d85db9378072c88b101bb76b647e9732dc1c363a1ce2a2d261400f19914e9936 Jul 30 20:51:50.634293 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:50.634251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" event={"ID":"41e69e04-a20b-4e2f-a9b3-3f2b48239d20","Type":"ContainerStarted","Data":"9b9ed80841e1b146d673729e21200d8f8acf4a3ef27dc697b9cc8501dd83ac2c"} Jul 30 20:51:50.634806 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:50.634335 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" event={"ID":"41e69e04-a20b-4e2f-a9b3-3f2b48239d20","Type":"ContainerStarted","Data":"d85db9378072c88b101bb76b647e9732dc1c363a1ce2a2d261400f19914e9936"} Jul 30 20:51:50.656020 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:50.655950 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" podStartSLOduration=1.655932228 podStartE2EDuration="1.655932228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:51:50.652809828 +0000 UTC m=+1849.117948188" watchObservedRunningTime="2026-07-30 20:51:50.655932228 +0000 UTC m=+1849.121070594" Jul 30 20:51:51.641392 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:51.641352 2575 generic.go:356] "Generic (PLEG): container finished" podID="41e69e04-a20b-4e2f-a9b3-3f2b48239d20" containerID="9b9ed80841e1b146d673729e21200d8f8acf4a3ef27dc697b9cc8501dd83ac2c" exitCode=0 Jul 30 20:51:51.641833 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:51.641440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" event={"ID":"41e69e04-a20b-4e2f-a9b3-3f2b48239d20","Type":"ContainerDied","Data":"9b9ed80841e1b146d673729e21200d8f8acf4a3ef27dc697b9cc8501dd83ac2c"} Jul 30 20:51:52.787547 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:52.787513 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" Jul 30 20:51:52.863166 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:52.863125 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e69e04-a20b-4e2f-a9b3-3f2b48239d20-maas-api-service-ca\") pod \"41e69e04-a20b-4e2f-a9b3-3f2b48239d20\" (UID: \"41e69e04-a20b-4e2f-a9b3-3f2b48239d20\") " Jul 30 20:51:52.863521 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:52.863494 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e69e04-a20b-4e2f-a9b3-3f2b48239d20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41e69e04-a20b-4e2f-a9b3-3f2b48239d20" (UID: "41e69e04-a20b-4e2f-a9b3-3f2b48239d20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:51:52.964833 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:52.964796 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e69e04-a20b-4e2f-a9b3-3f2b48239d20-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:53.654300 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:53.654269 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" Jul 30 20:51:53.654488 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:53.654305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q" event={"ID":"41e69e04-a20b-4e2f-a9b3-3f2b48239d20","Type":"ContainerDied","Data":"d85db9378072c88b101bb76b647e9732dc1c363a1ce2a2d261400f19914e9936"} Jul 30 20:51:53.654488 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:53.654353 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d85db9378072c88b101bb76b647e9732dc1c363a1ce2a2d261400f19914e9936" Jul 30 20:51:56.229305 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.229258 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts"] Jul 30 20:51:56.229891 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.229555 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" podUID="49230553-6792-4f96-a086-79bcc9ccfcd2" containerName="payload-processing" containerID="cri-o://9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318" gracePeriod=30 Jul 30 20:51:56.235505 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.235478 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws"] Jul 30 20:51:56.235726 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.235702 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" podUID="3b82588d-56c8-44d9-b619-e2cd811285bc" containerName="payload-pre-processing" containerID="cri-o://cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e" gracePeriod=30 Jul 30 20:51:56.451563 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.451469 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q"] Jul 30 20:51:56.455172 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.455139 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4c9be7-2pg9q"] Jul 30 20:51:56.493148 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.493019 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:56.523031 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.523005 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:56.600505 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.600473 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9tt6\" (UniqueName: \"kubernetes.io/projected/49230553-6792-4f96-a086-79bcc9ccfcd2-kube-api-access-t9tt6\") pod \"49230553-6792-4f96-a086-79bcc9ccfcd2\" (UID: \"49230553-6792-4f96-a086-79bcc9ccfcd2\") " Jul 30 20:51:56.600695 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.600543 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xwkc\" (UniqueName: \"kubernetes.io/projected/3b82588d-56c8-44d9-b619-e2cd811285bc-kube-api-access-7xwkc\") pod \"3b82588d-56c8-44d9-b619-e2cd811285bc\" (UID: \"3b82588d-56c8-44d9-b619-e2cd811285bc\") " Jul 30 20:51:56.600695 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.600597 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49230553-6792-4f96-a086-79bcc9ccfcd2-plugins-config-volume\") pod \"49230553-6792-4f96-a086-79bcc9ccfcd2\" (UID: \"49230553-6792-4f96-a086-79bcc9ccfcd2\") " Jul 30 20:51:56.600695 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.600627 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b82588d-56c8-44d9-b619-e2cd811285bc-plugins-config-volume\") pod \"3b82588d-56c8-44d9-b619-e2cd811285bc\" (UID: \"3b82588d-56c8-44d9-b619-e2cd811285bc\") " Jul 30 20:51:56.601175 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.601140 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b82588d-56c8-44d9-b619-e2cd811285bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b82588d-56c8-44d9-b619-e2cd811285bc" (UID: "3b82588d-56c8-44d9-b619-e2cd811285bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:51:56.601175 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.601158 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49230553-6792-4f96-a086-79bcc9ccfcd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49230553-6792-4f96-a086-79bcc9ccfcd2" (UID: "49230553-6792-4f96-a086-79bcc9ccfcd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:51:56.604149 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.604097 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b82588d-56c8-44d9-b619-e2cd811285bc-kube-api-access-7xwkc" (OuterVolumeSpecName: "kube-api-access-7xwkc") pod "3b82588d-56c8-44d9-b619-e2cd811285bc" (UID: "3b82588d-56c8-44d9-b619-e2cd811285bc"). InnerVolumeSpecName "kube-api-access-7xwkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:51:56.604617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.604581 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49230553-6792-4f96-a086-79bcc9ccfcd2-kube-api-access-t9tt6" (OuterVolumeSpecName: "kube-api-access-t9tt6") pod "49230553-6792-4f96-a086-79bcc9ccfcd2" (UID: "49230553-6792-4f96-a086-79bcc9ccfcd2"). InnerVolumeSpecName "kube-api-access-t9tt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:51:56.672526 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.672485 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b82588d-56c8-44d9-b619-e2cd811285bc" containerID="cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e" exitCode=0 Jul 30 20:51:56.672707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.672535 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" Jul 30 20:51:56.672707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.672574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" event={"ID":"3b82588d-56c8-44d9-b619-e2cd811285bc","Type":"ContainerDied","Data":"cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e"} Jul 30 20:51:56.672707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.672631 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws" event={"ID":"3b82588d-56c8-44d9-b619-e2cd811285bc","Type":"ContainerDied","Data":"7b28941f61afd0a52915e71c83f4b26a39f0cfaab457ec2912bdc2ef6c8e02f7"} Jul 30 20:51:56.672707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.672655 2575 scope.go:117] "RemoveContainer" containerID="cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e" Jul 30 20:51:56.674451 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.674417 2575 generic.go:356] "Generic (PLEG): container finished" podID="49230553-6792-4f96-a086-79bcc9ccfcd2" containerID="9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318" exitCode=0 Jul 30 20:51:56.674545 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.674454 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" Jul 30 20:51:56.674545 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.674495 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" event={"ID":"49230553-6792-4f96-a086-79bcc9ccfcd2","Type":"ContainerDied","Data":"9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318"} Jul 30 20:51:56.674545 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.674543 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts" event={"ID":"49230553-6792-4f96-a086-79bcc9ccfcd2","Type":"ContainerDied","Data":"eeea26de989e610fbbd89e0a9202f2da8aca07dacab75b4044d1b43929c1cf99"} Jul 30 20:51:56.682524 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.682360 2575 scope.go:117] "RemoveContainer" containerID="cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e" Jul 30 20:51:56.682830 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:51:56.682804 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e\": container with ID starting with cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e not found: ID does not exist" containerID="cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e" Jul 30 20:51:56.682909 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.682843 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e"} err="failed to get container status \"cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e\": rpc error: code = NotFound desc = could not find container \"cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e\": container with ID starting with cb08ed39af89c963ee30430deeed84d2cbf24b536004ee2cd41a7b780bfb5a6e not found: ID does not exist" Jul 30 20:51:56.682909 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.682867 2575 scope.go:117] "RemoveContainer" containerID="9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318" Jul 30 20:51:56.692211 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.692189 2575 scope.go:117] "RemoveContainer" containerID="9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318" Jul 30 20:51:56.692534 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:51:56.692513 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318\": container with ID starting with 9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318 not found: ID does not exist" containerID="9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318" Jul 30 20:51:56.692602 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.692547 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318"} err="failed to get container status \"9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318\": rpc error: code = NotFound desc = could not find container \"9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318\": container with ID starting with 9c09e36b7fd55e79c88be4b9824d25df07aac23b15e4752de4576abcfaf09318 not found: ID does not exist" Jul 30 20:51:56.701419 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.701393 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xwkc\" (UniqueName: \"kubernetes.io/projected/3b82588d-56c8-44d9-b619-e2cd811285bc-kube-api-access-7xwkc\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:56.701419 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.701418 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49230553-6792-4f96-a086-79bcc9ccfcd2-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:56.701419 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.701427 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b82588d-56c8-44d9-b619-e2cd811285bc-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:56.701656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.701436 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9tt6\" (UniqueName: \"kubernetes.io/projected/49230553-6792-4f96-a086-79bcc9ccfcd2-kube-api-access-t9tt6\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:51:56.714981 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.714946 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts"] Jul 30 20:51:56.721574 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.721541 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4c9be7-56c878c7fc-4fnts"] Jul 30 20:51:56.743112 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.743067 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws"] Jul 30 20:51:56.747077 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:56.746986 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4c9be7-7c8f9d94bd-768ws"] Jul 30 20:51:58.014272 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:58.014238 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b82588d-56c8-44d9-b619-e2cd811285bc" path="/var/lib/kubelet/pods/3b82588d-56c8-44d9-b619-e2cd811285bc/volumes" Jul 30 20:51:58.014630 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:58.014568 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e69e04-a20b-4e2f-a9b3-3f2b48239d20" path="/var/lib/kubelet/pods/41e69e04-a20b-4e2f-a9b3-3f2b48239d20/volumes" Jul 30 20:51:58.014885 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:51:58.014872 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49230553-6792-4f96-a086-79bcc9ccfcd2" path="/var/lib/kubelet/pods/49230553-6792-4f96-a086-79bcc9ccfcd2/volumes" Jul 30 20:52:01.529549 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529517 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx"] Jul 30 20:52:01.529969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529864 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b82588d-56c8-44d9-b619-e2cd811285bc" containerName="payload-pre-processing" Jul 30 20:52:01.529969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529876 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b82588d-56c8-44d9-b619-e2cd811285bc" containerName="payload-pre-processing" Jul 30 20:52:01.529969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529889 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e69e04-a20b-4e2f-a9b3-3f2b48239d20" containerName="revoke-keys" Jul 30 20:52:01.529969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529895 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e69e04-a20b-4e2f-a9b3-3f2b48239d20" containerName="revoke-keys" Jul 30 20:52:01.529969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529907 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49230553-6792-4f96-a086-79bcc9ccfcd2" containerName="payload-processing" Jul 30 20:52:01.529969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529913 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49230553-6792-4f96-a086-79bcc9ccfcd2" containerName="payload-processing" Jul 30 20:52:01.530212 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529974 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e69e04-a20b-4e2f-a9b3-3f2b48239d20" containerName="revoke-keys" Jul 30 20:52:01.530212 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529980 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49230553-6792-4f96-a086-79bcc9ccfcd2" containerName="payload-processing" Jul 30 20:52:01.530212 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.529988 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b82588d-56c8-44d9-b619-e2cd811285bc" containerName="payload-pre-processing" Jul 30 20:52:01.534616 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.534593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:01.537406 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.537383 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bfe7e24b-dockercfg-fbqq2\"" Jul 30 20:52:01.537752 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.537737 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bfe7e24b\"" Jul 30 20:52:01.546436 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.546414 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx"] Jul 30 20:52:01.583097 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.583061 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p"] Jul 30 20:52:01.586491 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.586473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:01.600222 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.600193 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p"] Jul 30 20:52:01.644220 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.644189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42144baf-0e45-4baf-9c73-3eb1a6155843-plugins-config-volume\") pod \"payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx\" (UID: \"42144baf-0e45-4baf-9c73-3eb1a6155843\") " pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:01.644387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.644234 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgxll\" (UniqueName: \"kubernetes.io/projected/42144baf-0e45-4baf-9c73-3eb1a6155843-kube-api-access-lgxll\") pod \"payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx\" (UID: \"42144baf-0e45-4baf-9c73-3eb1a6155843\") " pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:01.644387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.644270 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c67aa8-5e48-45c1-894e-975a7b418cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p\" (UID: \"54c67aa8-5e48-45c1-894e-975a7b418cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:01.644387 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.644336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5pm\" (UniqueName: \"kubernetes.io/projected/54c67aa8-5e48-45c1-894e-975a7b418cfe-kube-api-access-rl5pm\") pod \"payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p\" (UID: \"54c67aa8-5e48-45c1-894e-975a7b418cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:01.745017 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.744977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxll\" (UniqueName: \"kubernetes.io/projected/42144baf-0e45-4baf-9c73-3eb1a6155843-kube-api-access-lgxll\") pod \"payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx\" (UID: \"42144baf-0e45-4baf-9c73-3eb1a6155843\") " pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:01.745240 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.745037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c67aa8-5e48-45c1-894e-975a7b418cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p\" (UID: \"54c67aa8-5e48-45c1-894e-975a7b418cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:01.745240 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.745118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5pm\" (UniqueName: \"kubernetes.io/projected/54c67aa8-5e48-45c1-894e-975a7b418cfe-kube-api-access-rl5pm\") pod \"payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p\" (UID: \"54c67aa8-5e48-45c1-894e-975a7b418cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:01.745240 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.745181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42144baf-0e45-4baf-9c73-3eb1a6155843-plugins-config-volume\") pod \"payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx\" (UID: \"42144baf-0e45-4baf-9c73-3eb1a6155843\") " pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:01.745917 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.745891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42144baf-0e45-4baf-9c73-3eb1a6155843-plugins-config-volume\") pod \"payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx\" (UID: \"42144baf-0e45-4baf-9c73-3eb1a6155843\") " pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:01.745995 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.745933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c67aa8-5e48-45c1-894e-975a7b418cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p\" (UID: \"54c67aa8-5e48-45c1-894e-975a7b418cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:01.754561 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.754530 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxll\" (UniqueName: \"kubernetes.io/projected/42144baf-0e45-4baf-9c73-3eb1a6155843-kube-api-access-lgxll\") pod \"payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx\" (UID: \"42144baf-0e45-4baf-9c73-3eb1a6155843\") " pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:01.754673 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.754649 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5pm\" (UniqueName: \"kubernetes.io/projected/54c67aa8-5e48-45c1-894e-975a7b418cfe-kube-api-access-rl5pm\") pod \"payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p\" (UID: \"54c67aa8-5e48-45c1-894e-975a7b418cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:01.846457 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.846372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:01.896151 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:01.896115 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:02.014851 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:52:02.014755 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42144baf_0e45_4baf_9c73_3eb1a6155843.slice/crio-084bb305b17d4544c4f51355d0c9b093045684657b0bd8536a86e56a5460dea1 WatchSource:0}: Error finding container 084bb305b17d4544c4f51355d0c9b093045684657b0bd8536a86e56a5460dea1: Status 404 returned error can't find the container with id 084bb305b17d4544c4f51355d0c9b093045684657b0bd8536a86e56a5460dea1 Jul 30 20:52:02.019216 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.019193 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx"] Jul 30 20:52:02.065521 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.065495 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p"] Jul 30 20:52:02.068928 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:52:02.068893 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c67aa8_5e48_45c1_894e_975a7b418cfe.slice/crio-9e57204552c4e4f9b2059a5d2012e5e042289cc2017c0e6d1cad98deb4e91ec1 WatchSource:0}: Error finding container 9e57204552c4e4f9b2059a5d2012e5e042289cc2017c0e6d1cad98deb4e91ec1: Status 404 returned error can't find the container with id 9e57204552c4e4f9b2059a5d2012e5e042289cc2017c0e6d1cad98deb4e91ec1 Jul 30 20:52:02.713085 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.713020 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" event={"ID":"42144baf-0e45-4baf-9c73-3eb1a6155843","Type":"ContainerStarted","Data":"b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16"} Jul 30 20:52:02.713835 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.713105 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" event={"ID":"42144baf-0e45-4baf-9c73-3eb1a6155843","Type":"ContainerStarted","Data":"084bb305b17d4544c4f51355d0c9b093045684657b0bd8536a86e56a5460dea1"} Jul 30 20:52:02.714794 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.714770 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" event={"ID":"54c67aa8-5e48-45c1-894e-975a7b418cfe","Type":"ContainerStarted","Data":"526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101"} Jul 30 20:52:02.714906 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.714821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" event={"ID":"54c67aa8-5e48-45c1-894e-975a7b418cfe","Type":"ContainerStarted","Data":"9e57204552c4e4f9b2059a5d2012e5e042289cc2017c0e6d1cad98deb4e91ec1"} Jul 30 20:52:02.714906 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.714849 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:02.735609 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.735550 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" podStartSLOduration=1.7355312889999999 podStartE2EDuration="1.735531289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:52:02.732119165 +0000 UTC m=+1861.197257514" watchObservedRunningTime="2026-07-30 20:52:02.735531289 +0000 UTC m=+1861.200669643" Jul 30 20:52:02.754794 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:02.754731 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" podStartSLOduration=1.754715663 podStartE2EDuration="1.754715663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:52:02.753316711 +0000 UTC m=+1861.218455079" watchObservedRunningTime="2026-07-30 20:52:02.754715663 +0000 UTC m=+1861.219854010" Jul 30 20:52:03.725802 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:03.725761 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:13.728453 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:13.728419 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:14.735066 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:14.735032 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:31.352822 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.352727 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx"] Jul 30 20:52:31.353500 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.352989 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" podUID="42144baf-0e45-4baf-9c73-3eb1a6155843" containerName="payload-processing" containerID="cri-o://b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16" gracePeriod=30 Jul 30 20:52:31.358141 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.358107 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p"] Jul 30 20:52:31.358999 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.358958 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" podUID="54c67aa8-5e48-45c1-894e-975a7b418cfe" containerName="payload-pre-processing" containerID="cri-o://526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101" gracePeriod=30 Jul 30 20:52:31.643133 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.643104 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:31.646503 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.646476 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:31.720812 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.720773 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgxll\" (UniqueName: \"kubernetes.io/projected/42144baf-0e45-4baf-9c73-3eb1a6155843-kube-api-access-lgxll\") pod \"42144baf-0e45-4baf-9c73-3eb1a6155843\" (UID: \"42144baf-0e45-4baf-9c73-3eb1a6155843\") " Jul 30 20:52:31.720812 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.720814 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c67aa8-5e48-45c1-894e-975a7b418cfe-plugins-config-volume\") pod \"54c67aa8-5e48-45c1-894e-975a7b418cfe\" (UID: \"54c67aa8-5e48-45c1-894e-975a7b418cfe\") " Jul 30 20:52:31.721030 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.720873 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42144baf-0e45-4baf-9c73-3eb1a6155843-plugins-config-volume\") pod \"42144baf-0e45-4baf-9c73-3eb1a6155843\" (UID: \"42144baf-0e45-4baf-9c73-3eb1a6155843\") " Jul 30 20:52:31.721030 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.720933 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl5pm\" (UniqueName: \"kubernetes.io/projected/54c67aa8-5e48-45c1-894e-975a7b418cfe-kube-api-access-rl5pm\") pod \"54c67aa8-5e48-45c1-894e-975a7b418cfe\" (UID: \"54c67aa8-5e48-45c1-894e-975a7b418cfe\") " Jul 30 20:52:31.721275 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.721247 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c67aa8-5e48-45c1-894e-975a7b418cfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54c67aa8-5e48-45c1-894e-975a7b418cfe" (UID: "54c67aa8-5e48-45c1-894e-975a7b418cfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:52:31.721353 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.721304 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42144baf-0e45-4baf-9c73-3eb1a6155843-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42144baf-0e45-4baf-9c73-3eb1a6155843" (UID: "42144baf-0e45-4baf-9c73-3eb1a6155843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:52:31.723010 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.722990 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42144baf-0e45-4baf-9c73-3eb1a6155843-kube-api-access-lgxll" (OuterVolumeSpecName: "kube-api-access-lgxll") pod "42144baf-0e45-4baf-9c73-3eb1a6155843" (UID: "42144baf-0e45-4baf-9c73-3eb1a6155843"). InnerVolumeSpecName "kube-api-access-lgxll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:52:31.723101 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.723021 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c67aa8-5e48-45c1-894e-975a7b418cfe-kube-api-access-rl5pm" (OuterVolumeSpecName: "kube-api-access-rl5pm") pod "54c67aa8-5e48-45c1-894e-975a7b418cfe" (UID: "54c67aa8-5e48-45c1-894e-975a7b418cfe"). InnerVolumeSpecName "kube-api-access-rl5pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:52:31.822427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.822386 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5pm\" (UniqueName: \"kubernetes.io/projected/54c67aa8-5e48-45c1-894e-975a7b418cfe-kube-api-access-rl5pm\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:52:31.822427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.822419 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgxll\" (UniqueName: \"kubernetes.io/projected/42144baf-0e45-4baf-9c73-3eb1a6155843-kube-api-access-lgxll\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:52:31.822427 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.822432 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c67aa8-5e48-45c1-894e-975a7b418cfe-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:52:31.822661 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.822444 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42144baf-0e45-4baf-9c73-3eb1a6155843-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:52:31.889559 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.889458 2575 generic.go:356] "Generic (PLEG): container finished" podID="54c67aa8-5e48-45c1-894e-975a7b418cfe" containerID="526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101" exitCode=0 Jul 30 20:52:31.889559 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.889507 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" Jul 30 20:52:31.889559 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.889550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" event={"ID":"54c67aa8-5e48-45c1-894e-975a7b418cfe","Type":"ContainerDied","Data":"526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101"} Jul 30 20:52:31.889843 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.889605 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p" event={"ID":"54c67aa8-5e48-45c1-894e-975a7b418cfe","Type":"ContainerDied","Data":"9e57204552c4e4f9b2059a5d2012e5e042289cc2017c0e6d1cad98deb4e91ec1"} Jul 30 20:52:31.889843 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.889630 2575 scope.go:117] "RemoveContainer" containerID="526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101" Jul 30 20:52:31.891168 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.891134 2575 generic.go:356] "Generic (PLEG): container finished" podID="42144baf-0e45-4baf-9c73-3eb1a6155843" containerID="b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16" exitCode=0 Jul 30 20:52:31.891286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.891168 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" event={"ID":"42144baf-0e45-4baf-9c73-3eb1a6155843","Type":"ContainerDied","Data":"b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16"} Jul 30 20:52:31.891286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.891183 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" Jul 30 20:52:31.891286 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.891209 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx" event={"ID":"42144baf-0e45-4baf-9c73-3eb1a6155843","Type":"ContainerDied","Data":"084bb305b17d4544c4f51355d0c9b093045684657b0bd8536a86e56a5460dea1"} Jul 30 20:52:31.898867 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.898752 2575 scope.go:117] "RemoveContainer" containerID="526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101" Jul 30 20:52:31.899160 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:52:31.899105 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101\": container with ID starting with 526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101 not found: ID does not exist" containerID="526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101" Jul 30 20:52:31.899160 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.899141 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101"} err="failed to get container status \"526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101\": rpc error: code = NotFound desc = could not find container \"526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101\": container with ID starting with 526afd30f51a22c8ceb2a6bf872281eea1d8ef6e471ea97e3ecdac2a7d859101 not found: ID does not exist" Jul 30 20:52:31.899160 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.899161 2575 scope.go:117] "RemoveContainer" containerID="b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16" Jul 30 20:52:31.908389 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.908368 2575 scope.go:117] "RemoveContainer" containerID="b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16" Jul 30 20:52:31.908765 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:52:31.908744 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16\": container with ID starting with b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16 not found: ID does not exist" containerID="b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16" Jul 30 20:52:31.908833 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.908775 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16"} err="failed to get container status \"b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16\": rpc error: code = NotFound desc = could not find container \"b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16\": container with ID starting with b91cd7e41ab84affe2e1ea4d55009908c021e55a97a5036eeb758b87fc169a16 not found: ID does not exist" Jul 30 20:52:31.913262 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.913233 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p"] Jul 30 20:52:31.917881 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.917854 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bfe7e24b-67549b6b96-8g65p"] Jul 30 20:52:31.929219 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.929186 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx"] Jul 30 20:52:31.932980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:31.932948 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bfe7e24b-79d79b8c4-557sx"] Jul 30 20:52:32.012907 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:32.012867 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42144baf-0e45-4baf-9c73-3eb1a6155843" path="/var/lib/kubelet/pods/42144baf-0e45-4baf-9c73-3eb1a6155843/volumes" Jul 30 20:52:32.013294 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:32.013280 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c67aa8-5e48-45c1-894e-975a7b418cfe" path="/var/lib/kubelet/pods/54c67aa8-5e48-45c1-894e-975a7b418cfe/volumes" Jul 30 20:52:37.504645 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.504604 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv"] Jul 30 20:52:37.505034 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.505001 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42144baf-0e45-4baf-9c73-3eb1a6155843" containerName="payload-processing" Jul 30 20:52:37.505034 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.505016 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42144baf-0e45-4baf-9c73-3eb1a6155843" containerName="payload-processing" Jul 30 20:52:37.505034 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.505025 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c67aa8-5e48-45c1-894e-975a7b418cfe" containerName="payload-pre-processing" Jul 30 20:52:37.505034 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.505032 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c67aa8-5e48-45c1-894e-975a7b418cfe" containerName="payload-pre-processing" Jul 30 20:52:37.505248 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.505138 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42144baf-0e45-4baf-9c73-3eb1a6155843" containerName="payload-processing" Jul 30 20:52:37.505248 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.505154 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c67aa8-5e48-45c1-894e-975a7b418cfe" containerName="payload-pre-processing" Jul 30 20:52:37.508274 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.508253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:37.511222 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.511198 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4cbc80de\"" Jul 30 20:52:37.520133 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.520102 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv"] Jul 30 20:52:37.534221 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.534181 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld"] Jul 30 20:52:37.537671 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.537643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:37.540355 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.540330 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4cbc80de-dockercfg-2hrlb\"" Jul 30 20:52:37.549608 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.549540 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld"] Jul 30 20:52:37.576994 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.576957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv\" (UID: \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:37.577195 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.577021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrq55\" (UniqueName: \"kubernetes.io/projected/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-kube-api-access-vrq55\") pod \"payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv\" (UID: \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:37.678436 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.678383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrq55\" (UniqueName: \"kubernetes.io/projected/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-kube-api-access-vrq55\") pod \"payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv\" (UID: \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:37.678623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.678464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbwf\" (UniqueName: \"kubernetes.io/projected/1b891fe1-8add-4d01-911e-969d50a93d21-kube-api-access-6sbwf\") pod \"payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld\" (UID: \"1b891fe1-8add-4d01-911e-969d50a93d21\") " pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:37.678623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.678495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv\" (UID: \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:37.678623 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.678511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b891fe1-8add-4d01-911e-969d50a93d21-plugins-config-volume\") pod \"payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld\" (UID: \"1b891fe1-8add-4d01-911e-969d50a93d21\") " pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:37.679474 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.679448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv\" (UID: \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:37.688211 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.688176 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrq55\" (UniqueName: \"kubernetes.io/projected/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-kube-api-access-vrq55\") pod \"payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv\" (UID: \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:37.779483 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.779375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbwf\" (UniqueName: \"kubernetes.io/projected/1b891fe1-8add-4d01-911e-969d50a93d21-kube-api-access-6sbwf\") pod \"payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld\" (UID: \"1b891fe1-8add-4d01-911e-969d50a93d21\") " pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:37.779483 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.779436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b891fe1-8add-4d01-911e-969d50a93d21-plugins-config-volume\") pod \"payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld\" (UID: \"1b891fe1-8add-4d01-911e-969d50a93d21\") " pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:37.780263 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.780241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b891fe1-8add-4d01-911e-969d50a93d21-plugins-config-volume\") pod \"payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld\" (UID: \"1b891fe1-8add-4d01-911e-969d50a93d21\") " pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:37.791820 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.791781 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbwf\" (UniqueName: \"kubernetes.io/projected/1b891fe1-8add-4d01-911e-969d50a93d21-kube-api-access-6sbwf\") pod \"payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld\" (UID: \"1b891fe1-8add-4d01-911e-969d50a93d21\") " pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:37.820272 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.820238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:37.860086 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:37.859225 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:38.004109 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.003999 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv"] Jul 30 20:52:38.006132 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:52:38.006102 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb488e0_5f1c_496f_a6b6_5a5f4bc90965.slice/crio-f851b537f5601bc2d84ad5ae58352058ba386f2451209e9dd89bd00e5c8e693b WatchSource:0}: Error finding container f851b537f5601bc2d84ad5ae58352058ba386f2451209e9dd89bd00e5c8e693b: Status 404 returned error can't find the container with id f851b537f5601bc2d84ad5ae58352058ba386f2451209e9dd89bd00e5c8e693b Jul 30 20:52:38.034485 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.034403 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld"] Jul 30 20:52:38.039277 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:52:38.039244 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b891fe1_8add_4d01_911e_969d50a93d21.slice/crio-26b3bc0ea893e1b2826b58986a200d0a7b25d6344bf73ef517c9be7fe957a83d WatchSource:0}: Error finding container 26b3bc0ea893e1b2826b58986a200d0a7b25d6344bf73ef517c9be7fe957a83d: Status 404 returned error can't find the container with id 26b3bc0ea893e1b2826b58986a200d0a7b25d6344bf73ef517c9be7fe957a83d Jul 30 20:52:38.936952 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.936816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" event={"ID":"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965","Type":"ContainerStarted","Data":"2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee"} Jul 30 20:52:38.936952 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.936917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" event={"ID":"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965","Type":"ContainerStarted","Data":"f851b537f5601bc2d84ad5ae58352058ba386f2451209e9dd89bd00e5c8e693b"} Jul 30 20:52:38.961306 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.936985 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:38.961306 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.938609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" event={"ID":"1b891fe1-8add-4d01-911e-969d50a93d21","Type":"ContainerStarted","Data":"5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1"} Jul 30 20:52:38.961306 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.938664 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" event={"ID":"1b891fe1-8add-4d01-911e-969d50a93d21","Type":"ContainerStarted","Data":"26b3bc0ea893e1b2826b58986a200d0a7b25d6344bf73ef517c9be7fe957a83d"} Jul 30 20:52:38.961306 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.938697 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:52:38.961306 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.954717 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" podStartSLOduration=1.954696953 podStartE2EDuration="1.954696953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:52:38.954127431 +0000 UTC m=+1897.419265781" watchObservedRunningTime="2026-07-30 20:52:38.954696953 +0000 UTC m=+1897.419835303" Jul 30 20:52:38.971634 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:38.971570 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" podStartSLOduration=1.9715525120000001 podStartE2EDuration="1.971552512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:52:38.969652651 +0000 UTC m=+1897.434790991" watchObservedRunningTime="2026-07-30 20:52:38.971552512 +0000 UTC m=+1897.436690862" Jul 30 20:52:49.945768 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:49.945728 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:52:49.946235 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:52:49.945924 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:53:00.607927 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:00.607883 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj"] Jul 30 20:53:00.611522 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:00.611502 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" Jul 30 20:53:00.614176 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:00.614149 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 20:53:00.619220 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:00.619189 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj"] Jul 30 20:53:00.684885 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:00.684847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6892d7b1-a91f-4ed1-9a85-fdea91b6c96b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj\" (UID: \"6892d7b1-a91f-4ed1-9a85-fdea91b6c96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" Jul 30 20:53:00.785941 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:00.785902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6892d7b1-a91f-4ed1-9a85-fdea91b6c96b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj\" (UID: \"6892d7b1-a91f-4ed1-9a85-fdea91b6c96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" Jul 30 20:53:00.786582 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:00.786562 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6892d7b1-a91f-4ed1-9a85-fdea91b6c96b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj\" (UID: \"6892d7b1-a91f-4ed1-9a85-fdea91b6c96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" Jul 30 20:53:00.925195 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:00.925103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" Jul 30 20:53:01.069039 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:01.068990 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj"] Jul 30 20:53:01.069316 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:53:01.069294 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6892d7b1_a91f_4ed1_9a85_fdea91b6c96b.slice/crio-117fc35feb42fa7bb48a520b6b464380d2ed11c3367358e5b55127aede55dc87 WatchSource:0}: Error finding container 117fc35feb42fa7bb48a520b6b464380d2ed11c3367358e5b55127aede55dc87: Status 404 returned error can't find the container with id 117fc35feb42fa7bb48a520b6b464380d2ed11c3367358e5b55127aede55dc87 Jul 30 20:53:02.062825 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:02.062722 2575 generic.go:356] "Generic (PLEG): container finished" podID="6892d7b1-a91f-4ed1-9a85-fdea91b6c96b" containerID="0fcb1f95364a7611e6bf7023dba16b38a4ac4e6707ae14ba0592627d7f24c2ba" exitCode=0 Jul 30 20:53:02.062825 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:02.062807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" event={"ID":"6892d7b1-a91f-4ed1-9a85-fdea91b6c96b","Type":"ContainerDied","Data":"0fcb1f95364a7611e6bf7023dba16b38a4ac4e6707ae14ba0592627d7f24c2ba"} Jul 30 20:53:02.063252 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:02.062862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" event={"ID":"6892d7b1-a91f-4ed1-9a85-fdea91b6c96b","Type":"ContainerStarted","Data":"117fc35feb42fa7bb48a520b6b464380d2ed11c3367358e5b55127aede55dc87"} Jul 30 20:53:03.201624 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:03.201595 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" Jul 30 20:53:03.205166 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:03.205144 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6892d7b1-a91f-4ed1-9a85-fdea91b6c96b-maas-api-service-ca\") pod \"6892d7b1-a91f-4ed1-9a85-fdea91b6c96b\" (UID: \"6892d7b1-a91f-4ed1-9a85-fdea91b6c96b\") " Jul 30 20:53:03.205467 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:03.205445 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6892d7b1-a91f-4ed1-9a85-fdea91b6c96b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6892d7b1-a91f-4ed1-9a85-fdea91b6c96b" (UID: "6892d7b1-a91f-4ed1-9a85-fdea91b6c96b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:53:03.305779 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:03.305734 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6892d7b1-a91f-4ed1-9a85-fdea91b6c96b-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:53:04.075484 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:04.075450 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" Jul 30 20:53:04.075484 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:04.075473 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj" event={"ID":"6892d7b1-a91f-4ed1-9a85-fdea91b6c96b","Type":"ContainerDied","Data":"117fc35feb42fa7bb48a520b6b464380d2ed11c3367358e5b55127aede55dc87"} Jul 30 20:53:04.075700 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:04.075531 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="117fc35feb42fa7bb48a520b6b464380d2ed11c3367358e5b55127aede55dc87" Jul 30 20:53:07.503649 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.503609 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld"] Jul 30 20:53:07.504175 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.503877 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" podUID="1b891fe1-8add-4d01-911e-969d50a93d21" containerName="payload-processing" containerID="cri-o://5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1" gracePeriod=30 Jul 30 20:53:07.512160 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.512128 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv"] Jul 30 20:53:07.512410 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.512381 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" podUID="fcb488e0-5f1c-496f-a6b6-5a5f4bc90965" containerName="payload-pre-processing" containerID="cri-o://2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee" gracePeriod=30 Jul 30 20:53:07.737786 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.737705 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj"] Jul 30 20:53:07.740148 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.740078 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cbc80de-x8nwj"] Jul 30 20:53:07.803696 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.803653 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:53:07.806694 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.806665 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:53:07.840976 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.840940 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b891fe1-8add-4d01-911e-969d50a93d21-plugins-config-volume\") pod \"1b891fe1-8add-4d01-911e-969d50a93d21\" (UID: \"1b891fe1-8add-4d01-911e-969d50a93d21\") " Jul 30 20:53:07.841193 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.841016 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sbwf\" (UniqueName: \"kubernetes.io/projected/1b891fe1-8add-4d01-911e-969d50a93d21-kube-api-access-6sbwf\") pod \"1b891fe1-8add-4d01-911e-969d50a93d21\" (UID: \"1b891fe1-8add-4d01-911e-969d50a93d21\") " Jul 30 20:53:07.841193 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.841081 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-plugins-config-volume\") pod \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\" (UID: \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\") " Jul 30 20:53:07.841193 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.841134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrq55\" (UniqueName: \"kubernetes.io/projected/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-kube-api-access-vrq55\") pod \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\" (UID: \"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965\") " Jul 30 20:53:07.841497 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.841451 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b891fe1-8add-4d01-911e-969d50a93d21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b891fe1-8add-4d01-911e-969d50a93d21" (UID: "1b891fe1-8add-4d01-911e-969d50a93d21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:53:07.841563 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.841534 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fcb488e0-5f1c-496f-a6b6-5a5f4bc90965" (UID: "fcb488e0-5f1c-496f-a6b6-5a5f4bc90965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:53:07.843304 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.843272 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b891fe1-8add-4d01-911e-969d50a93d21-kube-api-access-6sbwf" (OuterVolumeSpecName: "kube-api-access-6sbwf") pod "1b891fe1-8add-4d01-911e-969d50a93d21" (UID: "1b891fe1-8add-4d01-911e-969d50a93d21"). InnerVolumeSpecName "kube-api-access-6sbwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:53:07.843430 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.843374 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-kube-api-access-vrq55" (OuterVolumeSpecName: "kube-api-access-vrq55") pod "fcb488e0-5f1c-496f-a6b6-5a5f4bc90965" (UID: "fcb488e0-5f1c-496f-a6b6-5a5f4bc90965"). InnerVolumeSpecName "kube-api-access-vrq55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:53:07.942409 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.942366 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b891fe1-8add-4d01-911e-969d50a93d21-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:53:07.942409 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.942411 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbwf\" (UniqueName: \"kubernetes.io/projected/1b891fe1-8add-4d01-911e-969d50a93d21-kube-api-access-6sbwf\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:53:07.942637 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.942425 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:53:07.942637 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:07.942437 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrq55\" (UniqueName: \"kubernetes.io/projected/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965-kube-api-access-vrq55\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:53:08.013307 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.013266 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6892d7b1-a91f-4ed1-9a85-fdea91b6c96b" path="/var/lib/kubelet/pods/6892d7b1-a91f-4ed1-9a85-fdea91b6c96b/volumes" Jul 30 20:53:08.099588 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.099557 2575 generic.go:356] "Generic (PLEG): container finished" podID="fcb488e0-5f1c-496f-a6b6-5a5f4bc90965" containerID="2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee" exitCode=0 Jul 30 20:53:08.099774 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.099619 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" Jul 30 20:53:08.099774 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.099628 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" event={"ID":"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965","Type":"ContainerDied","Data":"2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee"} Jul 30 20:53:08.099774 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.099674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv" event={"ID":"fcb488e0-5f1c-496f-a6b6-5a5f4bc90965","Type":"ContainerDied","Data":"f851b537f5601bc2d84ad5ae58352058ba386f2451209e9dd89bd00e5c8e693b"} Jul 30 20:53:08.099774 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.099705 2575 scope.go:117] "RemoveContainer" containerID="2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee" Jul 30 20:53:08.100958 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.100927 2575 generic.go:356] "Generic (PLEG): container finished" podID="1b891fe1-8add-4d01-911e-969d50a93d21" containerID="5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1" exitCode=0 Jul 30 20:53:08.101089 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.100986 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" Jul 30 20:53:08.101089 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.100981 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" event={"ID":"1b891fe1-8add-4d01-911e-969d50a93d21","Type":"ContainerDied","Data":"5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1"} Jul 30 20:53:08.101089 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.101044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld" event={"ID":"1b891fe1-8add-4d01-911e-969d50a93d21","Type":"ContainerDied","Data":"26b3bc0ea893e1b2826b58986a200d0a7b25d6344bf73ef517c9be7fe957a83d"} Jul 30 20:53:08.108997 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.108978 2575 scope.go:117] "RemoveContainer" containerID="2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee" Jul 30 20:53:08.109355 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:53:08.109327 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee\": container with ID starting with 2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee not found: ID does not exist" containerID="2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee" Jul 30 20:53:08.109446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.109365 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee"} err="failed to get container status \"2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee\": rpc error: code = NotFound desc = could not find container \"2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee\": container with ID starting with 2e84b462ab43d3dd4f050eccdaf0fba98ef4c92966ea43935fbd27a9e1a1b5ee not found: ID does not exist" Jul 30 20:53:08.109446 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.109387 2575 scope.go:117] "RemoveContainer" containerID="5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1" Jul 30 20:53:08.118463 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.118445 2575 scope.go:117] "RemoveContainer" containerID="5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1" Jul 30 20:53:08.118778 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:53:08.118749 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1\": container with ID starting with 5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1 not found: ID does not exist" containerID="5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1" Jul 30 20:53:08.118836 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.118786 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1"} err="failed to get container status \"5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1\": rpc error: code = NotFound desc = could not find container \"5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1\": container with ID starting with 5a05cd13a12e97ba2edcb6633554c08e3b012ffd48e89e27f7822dc3df4995a1 not found: ID does not exist" Jul 30 20:53:08.120859 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.120834 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv"] Jul 30 20:53:08.123670 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.123648 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cbc80de-77c4c4b95b-dm2fv"] Jul 30 20:53:08.133561 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.133530 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld"] Jul 30 20:53:08.140663 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:08.140635 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cbc80de-77b689ffdb-kzgld"] Jul 30 20:53:10.012877 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:10.012843 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b891fe1-8add-4d01-911e-969d50a93d21" path="/var/lib/kubelet/pods/1b891fe1-8add-4d01-911e-969d50a93d21/volumes" Jul 30 20:53:10.013252 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:53:10.013229 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb488e0-5f1c-496f-a6b6-5a5f4bc90965" path="/var/lib/kubelet/pods/fcb488e0-5f1c-496f-a6b6-5a5f4bc90965/volumes" Jul 30 20:55:10.948392 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948351 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp"] Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948704 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6892d7b1-a91f-4ed1-9a85-fdea91b6c96b" containerName="revoke-keys" Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948715 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6892d7b1-a91f-4ed1-9a85-fdea91b6c96b" containerName="revoke-keys" Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948730 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b891fe1-8add-4d01-911e-969d50a93d21" containerName="payload-processing" Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948735 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b891fe1-8add-4d01-911e-969d50a93d21" containerName="payload-processing" Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948752 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb488e0-5f1c-496f-a6b6-5a5f4bc90965" containerName="payload-pre-processing" Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948758 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb488e0-5f1c-496f-a6b6-5a5f4bc90965" containerName="payload-pre-processing" Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948813 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6892d7b1-a91f-4ed1-9a85-fdea91b6c96b" containerName="revoke-keys" Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948821 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb488e0-5f1c-496f-a6b6-5a5f4bc90965" containerName="payload-pre-processing" Jul 30 20:55:10.948889 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.948827 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b891fe1-8add-4d01-911e-969d50a93d21" containerName="payload-processing" Jul 30 20:55:10.952129 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.952106 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:10.955205 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.955186 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dd978c01\"" Jul 30 20:55:10.963945 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.963917 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp"] Jul 30 20:55:10.973390 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.973359 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj"] Jul 30 20:55:10.977202 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.977172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:10.982460 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.982326 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-dd978c01-dockercfg-p8zvn\"" Jul 30 20:55:10.989833 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.989746 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj"] Jul 30 20:55:10.992211 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.992179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtw5n\" (UniqueName: \"kubernetes.io/projected/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-kube-api-access-dtw5n\") pod \"payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp\" (UID: \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:10.992372 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.992223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tml\" (UniqueName: \"kubernetes.io/projected/defbe682-1fc6-402b-98c6-b028c377a3f0-kube-api-access-r9tml\") pod \"payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj\" (UID: \"defbe682-1fc6-402b-98c6-b028c377a3f0\") " pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:10.992445 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.992370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defbe682-1fc6-402b-98c6-b028c377a3f0-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj\" (UID: \"defbe682-1fc6-402b-98c6-b028c377a3f0\") " pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:10.992445 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:10.992425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp\" (UID: \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:11.092929 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.092895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defbe682-1fc6-402b-98c6-b028c377a3f0-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj\" (UID: \"defbe682-1fc6-402b-98c6-b028c377a3f0\") " pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:11.093160 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.092944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp\" (UID: \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:11.093160 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.092991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtw5n\" (UniqueName: \"kubernetes.io/projected/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-kube-api-access-dtw5n\") pod \"payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp\" (UID: \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:11.093160 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.093018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tml\" (UniqueName: \"kubernetes.io/projected/defbe682-1fc6-402b-98c6-b028c377a3f0-kube-api-access-r9tml\") pod \"payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj\" (UID: \"defbe682-1fc6-402b-98c6-b028c377a3f0\") " pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:11.093900 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.093875 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp\" (UID: \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:11.093951 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.093875 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defbe682-1fc6-402b-98c6-b028c377a3f0-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj\" (UID: \"defbe682-1fc6-402b-98c6-b028c377a3f0\") " pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:11.104121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.104093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tml\" (UniqueName: \"kubernetes.io/projected/defbe682-1fc6-402b-98c6-b028c377a3f0-kube-api-access-r9tml\") pod \"payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj\" (UID: \"defbe682-1fc6-402b-98c6-b028c377a3f0\") " pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:11.104259 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.104206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtw5n\" (UniqueName: \"kubernetes.io/projected/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-kube-api-access-dtw5n\") pod \"payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp\" (UID: \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:11.264302 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.264261 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:11.290617 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.290582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:11.431518 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.431488 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp"] Jul 30 20:55:11.433626 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:55:11.433599 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2aee44_b925_4d9b_8eb2_a6aaec79c7b4.slice/crio-da5eee70461e6900f9287c69ddfc2a444f21eca918f47546cbd4056a9bd2a458 WatchSource:0}: Error finding container da5eee70461e6900f9287c69ddfc2a444f21eca918f47546cbd4056a9bd2a458: Status 404 returned error can't find the container with id da5eee70461e6900f9287c69ddfc2a444f21eca918f47546cbd4056a9bd2a458 Jul 30 20:55:11.455797 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.455771 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj"] Jul 30 20:55:11.456473 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:55:11.456445 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefbe682_1fc6_402b_98c6_b028c377a3f0.slice/crio-ab7abb1172447b199915d0f29cef184dffc25620908d5dfde414d319182f9364 WatchSource:0}: Error finding container ab7abb1172447b199915d0f29cef184dffc25620908d5dfde414d319182f9364: Status 404 returned error can't find the container with id ab7abb1172447b199915d0f29cef184dffc25620908d5dfde414d319182f9364 Jul 30 20:55:11.798152 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.798115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" event={"ID":"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4","Type":"ContainerStarted","Data":"d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639"} Jul 30 20:55:11.798366 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.798175 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" event={"ID":"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4","Type":"ContainerStarted","Data":"da5eee70461e6900f9287c69ddfc2a444f21eca918f47546cbd4056a9bd2a458"} Jul 30 20:55:11.798366 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.798212 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:11.799949 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.799924 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" event={"ID":"defbe682-1fc6-402b-98c6-b028c377a3f0","Type":"ContainerStarted","Data":"401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9"} Jul 30 20:55:11.800086 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.799967 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" event={"ID":"defbe682-1fc6-402b-98c6-b028c377a3f0","Type":"ContainerStarted","Data":"ab7abb1172447b199915d0f29cef184dffc25620908d5dfde414d319182f9364"} Jul 30 20:55:11.800086 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.799988 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:11.825980 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.819624 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" podStartSLOduration=1.8196017599999998 podStartE2EDuration="1.81960176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:55:11.817617444 +0000 UTC m=+2050.282755791" watchObservedRunningTime="2026-07-30 20:55:11.81960176 +0000 UTC m=+2050.284740108" Jul 30 20:55:11.837368 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:11.837308 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" podStartSLOduration=1.837287453 podStartE2EDuration="1.837287453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:55:11.837024426 +0000 UTC m=+2050.302162788" watchObservedRunningTime="2026-07-30 20:55:11.837287453 +0000 UTC m=+2050.302425803" Jul 30 20:55:22.807759 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:22.807725 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:22.808264 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:22.807897 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:41.474475 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.474381 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl"] Jul 30 20:55:41.477850 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.477829 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" Jul 30 20:55:41.480562 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.480540 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 20:55:41.485521 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.485493 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl"] Jul 30 20:55:41.584431 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.584386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb00e7-b59e-4958-a429-79977e813840-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd978c01-wgspl\" (UID: \"6abb00e7-b59e-4958-a429-79977e813840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" Jul 30 20:55:41.685517 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.685470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb00e7-b59e-4958-a429-79977e813840-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd978c01-wgspl\" (UID: \"6abb00e7-b59e-4958-a429-79977e813840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" Jul 30 20:55:41.686296 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.686274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb00e7-b59e-4958-a429-79977e813840-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd978c01-wgspl\" (UID: \"6abb00e7-b59e-4958-a429-79977e813840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" Jul 30 20:55:41.791381 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.791294 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" Jul 30 20:55:41.935280 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.935224 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl"] Jul 30 20:55:41.937628 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:55:41.937597 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abb00e7_b59e_4958_a429_79977e813840.slice/crio-929fc9c3bdf775e4aa395391ec23f3342d38e53e6c3b4d015cf9eae2a6935677 WatchSource:0}: Error finding container 929fc9c3bdf775e4aa395391ec23f3342d38e53e6c3b4d015cf9eae2a6935677: Status 404 returned error can't find the container with id 929fc9c3bdf775e4aa395391ec23f3342d38e53e6c3b4d015cf9eae2a6935677 Jul 30 20:55:41.972824 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:41.972788 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" event={"ID":"6abb00e7-b59e-4958-a429-79977e813840","Type":"ContainerStarted","Data":"929fc9c3bdf775e4aa395391ec23f3342d38e53e6c3b4d015cf9eae2a6935677"} Jul 30 20:55:42.979663 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:42.979630 2575 generic.go:356] "Generic (PLEG): container finished" podID="6abb00e7-b59e-4958-a429-79977e813840" containerID="ff3bd624d098658df345254f3f399853a99c270c10173ac138919fd678dde054" exitCode=0 Jul 30 20:55:42.980114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:42.979686 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" event={"ID":"6abb00e7-b59e-4958-a429-79977e813840","Type":"ContainerDied","Data":"ff3bd624d098658df345254f3f399853a99c270c10173ac138919fd678dde054"} Jul 30 20:55:44.122795 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:44.122769 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" Jul 30 20:55:44.208260 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:44.208220 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb00e7-b59e-4958-a429-79977e813840-maas-api-service-ca\") pod \"6abb00e7-b59e-4958-a429-79977e813840\" (UID: \"6abb00e7-b59e-4958-a429-79977e813840\") " Jul 30 20:55:44.208604 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:44.208580 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abb00e7-b59e-4958-a429-79977e813840-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6abb00e7-b59e-4958-a429-79977e813840" (UID: "6abb00e7-b59e-4958-a429-79977e813840"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:55:44.308918 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:44.308829 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb00e7-b59e-4958-a429-79977e813840-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:55:44.993186 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:44.993150 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" event={"ID":"6abb00e7-b59e-4958-a429-79977e813840","Type":"ContainerDied","Data":"929fc9c3bdf775e4aa395391ec23f3342d38e53e6c3b4d015cf9eae2a6935677"} Jul 30 20:55:44.993186 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:44.993182 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl" Jul 30 20:55:44.993409 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:44.993204 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929fc9c3bdf775e4aa395391ec23f3342d38e53e6c3b4d015cf9eae2a6935677" Jul 30 20:55:52.093184 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.093137 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj"] Jul 30 20:55:52.093644 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.093433 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" podUID="defbe682-1fc6-402b-98c6-b028c377a3f0" containerName="payload-processing" containerID="cri-o://401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9" gracePeriod=30 Jul 30 20:55:52.110401 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.110371 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp"] Jul 30 20:55:52.110662 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.110630 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" podUID="ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4" containerName="payload-pre-processing" containerID="cri-o://d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639" gracePeriod=30 Jul 30 20:55:52.306027 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.305520 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl"] Jul 30 20:55:52.309395 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.309357 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd978c01-wgspl"] Jul 30 20:55:52.394498 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.394473 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:52.397114 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.397092 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:52.485148 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.485112 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defbe682-1fc6-402b-98c6-b028c377a3f0-plugins-config-volume\") pod \"defbe682-1fc6-402b-98c6-b028c377a3f0\" (UID: \"defbe682-1fc6-402b-98c6-b028c377a3f0\") " Jul 30 20:55:52.485343 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.485230 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-plugins-config-volume\") pod \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\" (UID: \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\") " Jul 30 20:55:52.485343 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.485289 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtw5n\" (UniqueName: \"kubernetes.io/projected/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-kube-api-access-dtw5n\") pod \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\" (UID: \"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4\") " Jul 30 20:55:52.485343 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.485314 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9tml\" (UniqueName: \"kubernetes.io/projected/defbe682-1fc6-402b-98c6-b028c377a3f0-kube-api-access-r9tml\") pod \"defbe682-1fc6-402b-98c6-b028c377a3f0\" (UID: \"defbe682-1fc6-402b-98c6-b028c377a3f0\") " Jul 30 20:55:52.485613 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.485592 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defbe682-1fc6-402b-98c6-b028c377a3f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "defbe682-1fc6-402b-98c6-b028c377a3f0" (UID: "defbe682-1fc6-402b-98c6-b028c377a3f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:55:52.485682 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.485656 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4" (UID: "ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:55:52.487393 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.487372 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-kube-api-access-dtw5n" (OuterVolumeSpecName: "kube-api-access-dtw5n") pod "ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4" (UID: "ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4"). InnerVolumeSpecName "kube-api-access-dtw5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:55:52.487457 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.487404 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defbe682-1fc6-402b-98c6-b028c377a3f0-kube-api-access-r9tml" (OuterVolumeSpecName: "kube-api-access-r9tml") pod "defbe682-1fc6-402b-98c6-b028c377a3f0" (UID: "defbe682-1fc6-402b-98c6-b028c377a3f0"). InnerVolumeSpecName "kube-api-access-r9tml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:55:52.585944 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.585901 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defbe682-1fc6-402b-98c6-b028c377a3f0-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:55:52.585944 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.585937 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:55:52.585944 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.585947 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtw5n\" (UniqueName: \"kubernetes.io/projected/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4-kube-api-access-dtw5n\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:55:52.585944 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:52.585956 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9tml\" (UniqueName: \"kubernetes.io/projected/defbe682-1fc6-402b-98c6-b028c377a3f0-kube-api-access-r9tml\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:55:53.044494 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.044459 2575 generic.go:356] "Generic (PLEG): container finished" podID="ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4" containerID="d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639" exitCode=0 Jul 30 20:55:53.044700 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.044503 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" Jul 30 20:55:53.044700 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.044538 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" event={"ID":"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4","Type":"ContainerDied","Data":"d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639"} Jul 30 20:55:53.044700 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.044596 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp" event={"ID":"ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4","Type":"ContainerDied","Data":"da5eee70461e6900f9287c69ddfc2a444f21eca918f47546cbd4056a9bd2a458"} Jul 30 20:55:53.044700 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.044627 2575 scope.go:117] "RemoveContainer" containerID="d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639" Jul 30 20:55:53.046465 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.046419 2575 generic.go:356] "Generic (PLEG): container finished" podID="defbe682-1fc6-402b-98c6-b028c377a3f0" containerID="401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9" exitCode=0 Jul 30 20:55:53.046618 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.046466 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" Jul 30 20:55:53.046618 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.046513 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" event={"ID":"defbe682-1fc6-402b-98c6-b028c377a3f0","Type":"ContainerDied","Data":"401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9"} Jul 30 20:55:53.046618 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.046562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj" event={"ID":"defbe682-1fc6-402b-98c6-b028c377a3f0","Type":"ContainerDied","Data":"ab7abb1172447b199915d0f29cef184dffc25620908d5dfde414d319182f9364"} Jul 30 20:55:53.055137 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.055114 2575 scope.go:117] "RemoveContainer" containerID="d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639" Jul 30 20:55:53.055590 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:55:53.055498 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639\": container with ID starting with d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639 not found: ID does not exist" containerID="d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639" Jul 30 20:55:53.055590 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.055527 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639"} err="failed to get container status \"d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639\": rpc error: code = NotFound desc = could not find container \"d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639\": container with ID starting with d71d3ffd4a59595f7a2bd83b26ee50d822517bef39066ba242c3cc060cac6639 not found: ID does not exist" Jul 30 20:55:53.055590 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.055546 2575 scope.go:117] "RemoveContainer" containerID="401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9" Jul 30 20:55:53.064766 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.064743 2575 scope.go:117] "RemoveContainer" containerID="401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9" Jul 30 20:55:53.065094 ip-10-0-140-21 kubenswrapper[2575]: E0730 20:55:53.065071 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9\": container with ID starting with 401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9 not found: ID does not exist" containerID="401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9" Jul 30 20:55:53.065165 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.065104 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9"} err="failed to get container status \"401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9\": rpc error: code = NotFound desc = could not find container \"401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9\": container with ID starting with 401e3366732905dcae6d7eb5fd25c7bec29ac349972289d99f5eca85f28609b9 not found: ID does not exist" Jul 30 20:55:53.072588 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.072554 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp"] Jul 30 20:55:53.076529 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.076461 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd978c01-759f845956-9trzp"] Jul 30 20:55:53.086730 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.086689 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj"] Jul 30 20:55:53.090568 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:53.090534 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd978c01-5445c8f64b-88vvj"] Jul 30 20:55:54.013548 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:54.013501 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6abb00e7-b59e-4958-a429-79977e813840" path="/var/lib/kubelet/pods/6abb00e7-b59e-4958-a429-79977e813840/volumes" Jul 30 20:55:54.014194 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:54.014102 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="defbe682-1fc6-402b-98c6-b028c377a3f0" path="/var/lib/kubelet/pods/defbe682-1fc6-402b-98c6-b028c377a3f0/volumes" Jul 30 20:55:54.014525 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:55:54.014502 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4" path="/var/lib/kubelet/pods/ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4/volumes" Jul 30 20:56:02.185111 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:56:02.185082 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:56:02.189975 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:56:02.189947 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 20:58:02.311012 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:02.310977 2575 scope.go:117] "RemoveContainer" containerID="9b9ed80841e1b146d673729e21200d8f8acf4a3ef27dc697b9cc8501dd83ac2c" Jul 30 20:58:02.319683 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:02.319664 2575 scope.go:117] "RemoveContainer" containerID="6519ca7b00709d18da5450d6673ca0d6466bb8bbcc248b9d4147a309b54e4474" Jul 30 20:58:19.947009 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.946966 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl"] Jul 30 20:58:19.947656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947508 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="defbe682-1fc6-402b-98c6-b028c377a3f0" containerName="payload-processing" Jul 30 20:58:19.947656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947529 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="defbe682-1fc6-402b-98c6-b028c377a3f0" containerName="payload-processing" Jul 30 20:58:19.947656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947552 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4" containerName="payload-pre-processing" Jul 30 20:58:19.947656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947559 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4" containerName="payload-pre-processing" Jul 30 20:58:19.947656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947599 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abb00e7-b59e-4958-a429-79977e813840" containerName="revoke-keys" Jul 30 20:58:19.947656 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947610 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abb00e7-b59e-4958-a429-79977e813840" containerName="revoke-keys" Jul 30 20:58:19.947969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947714 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2aee44-b925-4d9b-8eb2-a6aaec79c7b4" containerName="payload-pre-processing" Jul 30 20:58:19.947969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947731 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6abb00e7-b59e-4958-a429-79977e813840" containerName="revoke-keys" Jul 30 20:58:19.947969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.947744 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="defbe682-1fc6-402b-98c6-b028c377a3f0" containerName="payload-processing" Jul 30 20:58:19.951063 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.951030 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:19.956799 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.956777 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-78475f23\"" Jul 30 20:58:19.970093 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:19.970067 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl"] Jul 30 20:58:20.006851 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.006814 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m"] Jul 30 20:58:20.010416 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.010393 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.014041 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.014020 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-78475f23-dockercfg-7qpjd\"" Jul 30 20:58:20.041823 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.041798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m"] Jul 30 20:58:20.109969 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.109932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nmk\" (UniqueName: \"kubernetes.io/projected/a71593d5-d1d0-4ef7-a806-7a694cd7270f-kube-api-access-k4nmk\") pod \"payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl\" (UID: \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:20.110171 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.110076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txb2c\" (UniqueName: \"kubernetes.io/projected/51e60d11-5611-49c9-82a4-63dc14ec5fcb-kube-api-access-txb2c\") pod \"payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m\" (UID: \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\") " pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.110171 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.110160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71593d5-d1d0-4ef7-a806-7a694cd7270f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl\" (UID: \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:20.110257 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.110182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e60d11-5611-49c9-82a4-63dc14ec5fcb-plugins-config-volume\") pod \"payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m\" (UID: \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\") " pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.211120 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.211094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txb2c\" (UniqueName: \"kubernetes.io/projected/51e60d11-5611-49c9-82a4-63dc14ec5fcb-kube-api-access-txb2c\") pod \"payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m\" (UID: \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\") " pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.211277 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.211146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71593d5-d1d0-4ef7-a806-7a694cd7270f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl\" (UID: \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:20.211277 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.211171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e60d11-5611-49c9-82a4-63dc14ec5fcb-plugins-config-volume\") pod \"payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m\" (UID: \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\") " pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.211363 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.211304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nmk\" (UniqueName: \"kubernetes.io/projected/a71593d5-d1d0-4ef7-a806-7a694cd7270f-kube-api-access-k4nmk\") pod \"payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl\" (UID: \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:20.211887 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.211866 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71593d5-d1d0-4ef7-a806-7a694cd7270f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl\" (UID: \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:20.211962 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.211946 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e60d11-5611-49c9-82a4-63dc14ec5fcb-plugins-config-volume\") pod \"payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m\" (UID: \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\") " pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.220752 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.220724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nmk\" (UniqueName: \"kubernetes.io/projected/a71593d5-d1d0-4ef7-a806-7a694cd7270f-kube-api-access-k4nmk\") pod \"payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl\" (UID: \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:20.221116 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.221093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txb2c\" (UniqueName: \"kubernetes.io/projected/51e60d11-5611-49c9-82a4-63dc14ec5fcb-kube-api-access-txb2c\") pod \"payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m\" (UID: \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\") " pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.260802 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.260767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:20.321671 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.321609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.416648 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.416612 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl"] Jul 30 20:58:20.417149 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:58:20.417125 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71593d5_d1d0_4ef7_a806_7a694cd7270f.slice/crio-990b1536073121c4f3a0ca8f80ae1920614edd5976745c921ab7eaece720cde9 WatchSource:0}: Error finding container 990b1536073121c4f3a0ca8f80ae1920614edd5976745c921ab7eaece720cde9: Status 404 returned error can't find the container with id 990b1536073121c4f3a0ca8f80ae1920614edd5976745c921ab7eaece720cde9 Jul 30 20:58:20.480002 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.479973 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m"] Jul 30 20:58:20.481967 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:58:20.481942 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e60d11_5611_49c9_82a4_63dc14ec5fcb.slice/crio-6e90e8f88013826f73a0bbf7088f915293ac52c06294ecab9c071f26a54670f7 WatchSource:0}: Error finding container 6e90e8f88013826f73a0bbf7088f915293ac52c06294ecab9c071f26a54670f7: Status 404 returned error can't find the container with id 6e90e8f88013826f73a0bbf7088f915293ac52c06294ecab9c071f26a54670f7 Jul 30 20:58:20.852736 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.852625 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" event={"ID":"51e60d11-5611-49c9-82a4-63dc14ec5fcb","Type":"ContainerStarted","Data":"b4a1ebf9777d1269f0913b3192d2a380c80594108d3c7a713a5659254b8d4b27"} Jul 30 20:58:20.852736 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.852698 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" event={"ID":"51e60d11-5611-49c9-82a4-63dc14ec5fcb","Type":"ContainerStarted","Data":"6e90e8f88013826f73a0bbf7088f915293ac52c06294ecab9c071f26a54670f7"} Jul 30 20:58:20.852736 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.852729 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:20.854200 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.854173 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" event={"ID":"a71593d5-d1d0-4ef7-a806-7a694cd7270f","Type":"ContainerStarted","Data":"37034a4288b218527f347db98e7a3a40318da8212d8712ec55e0cce6d3c5a4ab"} Jul 30 20:58:20.854331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.854218 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:58:20.854331 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.854235 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" event={"ID":"a71593d5-d1d0-4ef7-a806-7a694cd7270f","Type":"ContainerStarted","Data":"990b1536073121c4f3a0ca8f80ae1920614edd5976745c921ab7eaece720cde9"} Jul 30 20:58:20.872877 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.872822 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" podStartSLOduration=1.872805317 podStartE2EDuration="1.872805317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:58:20.871093892 +0000 UTC m=+2239.336232440" watchObservedRunningTime="2026-07-30 20:58:20.872805317 +0000 UTC m=+2239.337943664" Jul 30 20:58:20.888707 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:20.888662 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" podStartSLOduration=1.888648462 podStartE2EDuration="1.888648462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:58:20.886849879 +0000 UTC m=+2239.351988227" watchObservedRunningTime="2026-07-30 20:58:20.888648462 +0000 UTC m=+2239.353786810" Jul 30 20:58:31.862257 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:31.862165 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:58:31.862715 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:58:31.862262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:59:02.345444 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:02.345409 2575 scope.go:117] "RemoveContainer" containerID="0fcb1f95364a7611e6bf7023dba16b38a4ac4e6707ae14ba0592627d7f24c2ba" Jul 30 20:59:22.661301 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:22.661260 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt"] Jul 30 20:59:22.664767 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:22.664745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" Jul 30 20:59:22.667586 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:22.667565 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 20:59:22.674271 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:22.674244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt"] Jul 30 20:59:22.757906 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:22.757869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2754e29-b98f-4250-9ea6-59c92d4a5cfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-78475f23-p77kt\" (UID: \"c2754e29-b98f-4250-9ea6-59c92d4a5cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" Jul 30 20:59:22.859329 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:22.859284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2754e29-b98f-4250-9ea6-59c92d4a5cfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-78475f23-p77kt\" (UID: \"c2754e29-b98f-4250-9ea6-59c92d4a5cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" Jul 30 20:59:22.860042 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:22.860012 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2754e29-b98f-4250-9ea6-59c92d4a5cfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-78475f23-p77kt\" (UID: \"c2754e29-b98f-4250-9ea6-59c92d4a5cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" Jul 30 20:59:22.976928 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:22.976890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" Jul 30 20:59:23.129668 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:23.129637 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt"] Jul 30 20:59:23.131465 ip-10-0-140-21 kubenswrapper[2575]: W0730 20:59:23.131439 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2754e29_b98f_4250_9ea6_59c92d4a5cfe.slice/crio-747bb1c532b7e4152f13fd0983a2ae562d02219fac07c112f3270ef3e10013ac WatchSource:0}: Error finding container 747bb1c532b7e4152f13fd0983a2ae562d02219fac07c112f3270ef3e10013ac: Status 404 returned error can't find the container with id 747bb1c532b7e4152f13fd0983a2ae562d02219fac07c112f3270ef3e10013ac Jul 30 20:59:23.221935 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:23.221901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" event={"ID":"c2754e29-b98f-4250-9ea6-59c92d4a5cfe","Type":"ContainerStarted","Data":"747bb1c532b7e4152f13fd0983a2ae562d02219fac07c112f3270ef3e10013ac"} Jul 30 20:59:24.230988 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:24.230954 2575 generic.go:356] "Generic (PLEG): container finished" podID="c2754e29-b98f-4250-9ea6-59c92d4a5cfe" containerID="e7db63af1fc59cdfc1a4dada7559c2aad6a85c506a338cff136d716a92df8927" exitCode=0 Jul 30 20:59:24.231546 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:24.231048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" event={"ID":"c2754e29-b98f-4250-9ea6-59c92d4a5cfe","Type":"ContainerDied","Data":"e7db63af1fc59cdfc1a4dada7559c2aad6a85c506a338cff136d716a92df8927"} Jul 30 20:59:25.372165 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:25.372139 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" Jul 30 20:59:25.484348 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:25.484306 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2754e29-b98f-4250-9ea6-59c92d4a5cfe-maas-api-service-ca\") pod \"c2754e29-b98f-4250-9ea6-59c92d4a5cfe\" (UID: \"c2754e29-b98f-4250-9ea6-59c92d4a5cfe\") " Jul 30 20:59:25.484777 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:25.484752 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2754e29-b98f-4250-9ea6-59c92d4a5cfe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2754e29-b98f-4250-9ea6-59c92d4a5cfe" (UID: "c2754e29-b98f-4250-9ea6-59c92d4a5cfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:59:25.585168 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:25.585044 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2754e29-b98f-4250-9ea6-59c92d4a5cfe-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:59:26.243703 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:26.243673 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" Jul 30 20:59:26.243872 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:26.243713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt" event={"ID":"c2754e29-b98f-4250-9ea6-59c92d4a5cfe","Type":"ContainerDied","Data":"747bb1c532b7e4152f13fd0983a2ae562d02219fac07c112f3270ef3e10013ac"} Jul 30 20:59:26.243872 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:26.243759 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="747bb1c532b7e4152f13fd0983a2ae562d02219fac07c112f3270ef3e10013ac" Jul 30 20:59:38.167959 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.167917 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m"] Jul 30 20:59:38.168475 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.168214 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" podUID="51e60d11-5611-49c9-82a4-63dc14ec5fcb" containerName="payload-processing" containerID="cri-o://b4a1ebf9777d1269f0913b3192d2a380c80594108d3c7a713a5659254b8d4b27" gracePeriod=30 Jul 30 20:59:38.175869 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.175829 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl"] Jul 30 20:59:38.176247 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.176151 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" podUID="a71593d5-d1d0-4ef7-a806-7a694cd7270f" containerName="payload-pre-processing" containerID="cri-o://37034a4288b218527f347db98e7a3a40318da8212d8712ec55e0cce6d3c5a4ab" gracePeriod=30 Jul 30 20:59:38.309709 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.309667 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt"] Jul 30 20:59:38.314499 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.314458 2575 generic.go:356] "Generic (PLEG): container finished" podID="51e60d11-5611-49c9-82a4-63dc14ec5fcb" containerID="b4a1ebf9777d1269f0913b3192d2a380c80594108d3c7a713a5659254b8d4b27" exitCode=0 Jul 30 20:59:38.314678 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.314523 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" event={"ID":"51e60d11-5611-49c9-82a4-63dc14ec5fcb","Type":"ContainerDied","Data":"b4a1ebf9777d1269f0913b3192d2a380c80594108d3c7a713a5659254b8d4b27"} Jul 30 20:59:38.315016 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.314991 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-78475f23-p77kt"] Jul 30 20:59:38.316514 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.316491 2575 generic.go:356] "Generic (PLEG): container finished" podID="a71593d5-d1d0-4ef7-a806-7a694cd7270f" containerID="37034a4288b218527f347db98e7a3a40318da8212d8712ec55e0cce6d3c5a4ab" exitCode=0 Jul 30 20:59:38.316634 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.316562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" event={"ID":"a71593d5-d1d0-4ef7-a806-7a694cd7270f","Type":"ContainerDied","Data":"37034a4288b218527f347db98e7a3a40318da8212d8712ec55e0cce6d3c5a4ab"} Jul 30 20:59:38.456648 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.456617 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:59:38.477318 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.477289 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:59:38.478361 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.478345 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e60d11-5611-49c9-82a4-63dc14ec5fcb-plugins-config-volume\") pod \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\" (UID: \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\") " Jul 30 20:59:38.478429 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.478387 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4nmk\" (UniqueName: \"kubernetes.io/projected/a71593d5-d1d0-4ef7-a806-7a694cd7270f-kube-api-access-k4nmk\") pod \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\" (UID: \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\") " Jul 30 20:59:38.478429 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.478420 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txb2c\" (UniqueName: \"kubernetes.io/projected/51e60d11-5611-49c9-82a4-63dc14ec5fcb-kube-api-access-txb2c\") pod \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\" (UID: \"51e60d11-5611-49c9-82a4-63dc14ec5fcb\") " Jul 30 20:59:38.478509 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.478472 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71593d5-d1d0-4ef7-a806-7a694cd7270f-plugins-config-volume\") pod \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\" (UID: \"a71593d5-d1d0-4ef7-a806-7a694cd7270f\") " Jul 30 20:59:38.478891 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.478860 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e60d11-5611-49c9-82a4-63dc14ec5fcb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51e60d11-5611-49c9-82a4-63dc14ec5fcb" (UID: "51e60d11-5611-49c9-82a4-63dc14ec5fcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:59:38.479002 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.478903 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71593d5-d1d0-4ef7-a806-7a694cd7270f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a71593d5-d1d0-4ef7-a806-7a694cd7270f" (UID: "a71593d5-d1d0-4ef7-a806-7a694cd7270f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:59:38.480564 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.480540 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71593d5-d1d0-4ef7-a806-7a694cd7270f-kube-api-access-k4nmk" (OuterVolumeSpecName: "kube-api-access-k4nmk") pod "a71593d5-d1d0-4ef7-a806-7a694cd7270f" (UID: "a71593d5-d1d0-4ef7-a806-7a694cd7270f"). InnerVolumeSpecName "kube-api-access-k4nmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:59:38.480564 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.480555 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e60d11-5611-49c9-82a4-63dc14ec5fcb-kube-api-access-txb2c" (OuterVolumeSpecName: "kube-api-access-txb2c") pod "51e60d11-5611-49c9-82a4-63dc14ec5fcb" (UID: "51e60d11-5611-49c9-82a4-63dc14ec5fcb"). InnerVolumeSpecName "kube-api-access-txb2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:59:38.579492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.579455 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71593d5-d1d0-4ef7-a806-7a694cd7270f-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:59:38.579492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.579486 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e60d11-5611-49c9-82a4-63dc14ec5fcb-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:59:38.579492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.579496 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4nmk\" (UniqueName: \"kubernetes.io/projected/a71593d5-d1d0-4ef7-a806-7a694cd7270f-kube-api-access-k4nmk\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:59:38.579492 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:38.579504 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txb2c\" (UniqueName: \"kubernetes.io/projected/51e60d11-5611-49c9-82a4-63dc14ec5fcb-kube-api-access-txb2c\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 20:59:39.324121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.324087 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" Jul 30 20:59:39.324121 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.324103 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m" event={"ID":"51e60d11-5611-49c9-82a4-63dc14ec5fcb","Type":"ContainerDied","Data":"6e90e8f88013826f73a0bbf7088f915293ac52c06294ecab9c071f26a54670f7"} Jul 30 20:59:39.324663 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.324179 2575 scope.go:117] "RemoveContainer" containerID="b4a1ebf9777d1269f0913b3192d2a380c80594108d3c7a713a5659254b8d4b27" Jul 30 20:59:39.325588 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.325495 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" event={"ID":"a71593d5-d1d0-4ef7-a806-7a694cd7270f","Type":"ContainerDied","Data":"990b1536073121c4f3a0ca8f80ae1920614edd5976745c921ab7eaece720cde9"} Jul 30 20:59:39.325588 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.325505 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl" Jul 30 20:59:39.335368 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.335273 2575 scope.go:117] "RemoveContainer" containerID="37034a4288b218527f347db98e7a3a40318da8212d8712ec55e0cce6d3c5a4ab" Jul 30 20:59:39.350000 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.349964 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m"] Jul 30 20:59:39.353755 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.353724 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-78475f23-666898bdb6-7lq9m"] Jul 30 20:59:39.365630 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.365584 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl"] Jul 30 20:59:39.369543 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:39.369507 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-78475f23-7579dbcf6-fg8sl"] Jul 30 20:59:40.014172 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:40.014134 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e60d11-5611-49c9-82a4-63dc14ec5fcb" path="/var/lib/kubelet/pods/51e60d11-5611-49c9-82a4-63dc14ec5fcb/volumes" Jul 30 20:59:40.014535 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:40.014520 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a71593d5-d1d0-4ef7-a806-7a694cd7270f" path="/var/lib/kubelet/pods/a71593d5-d1d0-4ef7-a806-7a694cd7270f/volumes" Jul 30 20:59:40.014826 ip-10-0-140-21 kubenswrapper[2575]: I0730 20:59:40.014814 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2754e29-b98f-4250-9ea6-59c92d4a5cfe" path="/var/lib/kubelet/pods/c2754e29-b98f-4250-9ea6-59c92d4a5cfe/volumes" Jul 30 21:00:00.136793 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.136706 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr"] Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137109 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71593d5-d1d0-4ef7-a806-7a694cd7270f" containerName="payload-pre-processing" Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137122 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71593d5-d1d0-4ef7-a806-7a694cd7270f" containerName="payload-pre-processing" Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137136 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2754e29-b98f-4250-9ea6-59c92d4a5cfe" containerName="revoke-keys" Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137142 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2754e29-b98f-4250-9ea6-59c92d4a5cfe" containerName="revoke-keys" Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137151 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e60d11-5611-49c9-82a4-63dc14ec5fcb" containerName="payload-processing" Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137157 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e60d11-5611-49c9-82a4-63dc14ec5fcb" containerName="payload-processing" Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137225 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71593d5-d1d0-4ef7-a806-7a694cd7270f" containerName="payload-pre-processing" Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137234 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e60d11-5611-49c9-82a4-63dc14ec5fcb" containerName="payload-processing" Jul 30 21:00:00.137241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.137242 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2754e29-b98f-4250-9ea6-59c92d4a5cfe" containerName="revoke-keys" Jul 30 21:00:00.141684 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.141660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr" Jul 30 21:00:00.144693 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.144666 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:00:00.147497 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.147471 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr"] Jul 30 21:00:00.153951 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.153929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr" Jul 30 21:00:00.293622 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.293592 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr"] Jul 30 21:00:00.295313 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:00:00.295283 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16fe61d_707f_4b11_afc0_f237b2e6d1ec.slice/crio-498b0b0b2c98da0e7d545d8672f9f1ca865f68e8de2477957bcbb54d63e92f64 WatchSource:0}: Error finding container 498b0b0b2c98da0e7d545d8672f9f1ca865f68e8de2477957bcbb54d63e92f64: Status 404 returned error can't find the container with id 498b0b0b2c98da0e7d545d8672f9f1ca865f68e8de2477957bcbb54d63e92f64 Jul 30 21:00:00.444131 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:00.444020 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr" event={"ID":"a16fe61d-707f-4b11-afc0-f237b2e6d1ec","Type":"ContainerStarted","Data":"498b0b0b2c98da0e7d545d8672f9f1ca865f68e8de2477957bcbb54d63e92f64"} Jul 30 21:00:01.451307 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:01.451268 2575 generic.go:356] "Generic (PLEG): container finished" podID="a16fe61d-707f-4b11-afc0-f237b2e6d1ec" containerID="b2d664badce06c1f3f45ae35b9363540bf5ed725a9c9fdeb3ad0d555343e95cb" exitCode=0 Jul 30 21:00:01.451700 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:01.451360 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr" event={"ID":"a16fe61d-707f-4b11-afc0-f237b2e6d1ec","Type":"ContainerDied","Data":"b2d664badce06c1f3f45ae35b9363540bf5ed725a9c9fdeb3ad0d555343e95cb"} Jul 30 21:00:02.595973 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:02.595938 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr" Jul 30 21:00:03.463720 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:03.463686 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr" Jul 30 21:00:03.463908 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:03.463717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757420-6k5tr" event={"ID":"a16fe61d-707f-4b11-afc0-f237b2e6d1ec","Type":"ContainerDied","Data":"498b0b0b2c98da0e7d545d8672f9f1ca865f68e8de2477957bcbb54d63e92f64"} Jul 30 21:00:03.463908 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:00:03.463766 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="498b0b0b2c98da0e7d545d8672f9f1ca865f68e8de2477957bcbb54d63e92f64" Jul 30 21:01:02.219846 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:02.219816 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:01:02.228586 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:02.228552 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:01:40.318794 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.318708 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5"] Jul 30 21:01:40.319307 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.319090 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a16fe61d-707f-4b11-afc0-f237b2e6d1ec" containerName="cleanup" Jul 30 21:01:40.319307 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.319103 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a16fe61d-707f-4b11-afc0-f237b2e6d1ec" containerName="cleanup" Jul 30 21:01:40.319307 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.319163 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a16fe61d-707f-4b11-afc0-f237b2e6d1ec" containerName="cleanup" Jul 30 21:01:40.322101 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.322084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:40.328469 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.328441 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-72f733\"" Jul 30 21:01:40.339035 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.339001 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5"] Jul 30 21:01:40.385984 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.385948 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg"] Jul 30 21:01:40.389524 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.389500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.394581 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.394561 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-72f733-dockercfg-pzx8t\"" Jul 30 21:01:40.410049 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.410008 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg"] Jul 30 21:01:40.464433 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.464396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451f3bb-f5a1-4211-82c1-36679047eb56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5\" (UID: \"2451f3bb-f5a1-4211-82c1-36679047eb56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:40.464609 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.464489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6k4\" (UniqueName: \"kubernetes.io/projected/2451f3bb-f5a1-4211-82c1-36679047eb56-kube-api-access-jv6k4\") pod \"payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5\" (UID: \"2451f3bb-f5a1-4211-82c1-36679047eb56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:40.464609 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.464525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a08a0e-8f23-47a3-a71b-e09d01fe503d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg\" (UID: \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.464777 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.464760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcs8g\" (UniqueName: \"kubernetes.io/projected/33a08a0e-8f23-47a3-a71b-e09d01fe503d-kube-api-access-tcs8g\") pod \"payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg\" (UID: \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.566034 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.565998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451f3bb-f5a1-4211-82c1-36679047eb56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5\" (UID: \"2451f3bb-f5a1-4211-82c1-36679047eb56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:40.566034 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.566045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6k4\" (UniqueName: \"kubernetes.io/projected/2451f3bb-f5a1-4211-82c1-36679047eb56-kube-api-access-jv6k4\") pod \"payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5\" (UID: \"2451f3bb-f5a1-4211-82c1-36679047eb56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:40.566374 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.566084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a08a0e-8f23-47a3-a71b-e09d01fe503d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg\" (UID: \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.566374 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.566165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcs8g\" (UniqueName: \"kubernetes.io/projected/33a08a0e-8f23-47a3-a71b-e09d01fe503d-kube-api-access-tcs8g\") pod \"payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg\" (UID: \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.567030 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.567007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451f3bb-f5a1-4211-82c1-36679047eb56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5\" (UID: \"2451f3bb-f5a1-4211-82c1-36679047eb56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:40.567113 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.567027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a08a0e-8f23-47a3-a71b-e09d01fe503d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg\" (UID: \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.574699 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.574632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcs8g\" (UniqueName: \"kubernetes.io/projected/33a08a0e-8f23-47a3-a71b-e09d01fe503d-kube-api-access-tcs8g\") pod \"payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg\" (UID: \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.574824 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.574805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6k4\" (UniqueName: \"kubernetes.io/projected/2451f3bb-f5a1-4211-82c1-36679047eb56-kube-api-access-jv6k4\") pod \"payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5\" (UID: \"2451f3bb-f5a1-4211-82c1-36679047eb56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:40.632966 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.632925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:40.699889 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.699837 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.794574 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.794543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5"] Jul 30 21:01:40.796200 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:01:40.796171 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2451f3bb_f5a1_4211_82c1_36679047eb56.slice/crio-58c061f4e323b7e359bbd85fff5f19d77884b2699610ea5b8a6705e1a53ab5f3 WatchSource:0}: Error finding container 58c061f4e323b7e359bbd85fff5f19d77884b2699610ea5b8a6705e1a53ab5f3: Status 404 returned error can't find the container with id 58c061f4e323b7e359bbd85fff5f19d77884b2699610ea5b8a6705e1a53ab5f3 Jul 30 21:01:40.858259 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.858234 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg"] Jul 30 21:01:40.860151 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:01:40.860122 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a08a0e_8f23_47a3_a71b_e09d01fe503d.slice/crio-e82cf18701017b7d50b2a1ac33e8e768cf8b215e479b83a577cbce87666211dc WatchSource:0}: Error finding container e82cf18701017b7d50b2a1ac33e8e768cf8b215e479b83a577cbce87666211dc: Status 404 returned error can't find the container with id e82cf18701017b7d50b2a1ac33e8e768cf8b215e479b83a577cbce87666211dc Jul 30 21:01:40.991475 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.991396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" event={"ID":"33a08a0e-8f23-47a3-a71b-e09d01fe503d","Type":"ContainerStarted","Data":"5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f"} Jul 30 21:01:40.991687 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.991500 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:01:40.991687 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.991528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" event={"ID":"33a08a0e-8f23-47a3-a71b-e09d01fe503d","Type":"ContainerStarted","Data":"e82cf18701017b7d50b2a1ac33e8e768cf8b215e479b83a577cbce87666211dc"} Jul 30 21:01:40.993104 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.993049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" event={"ID":"2451f3bb-f5a1-4211-82c1-36679047eb56","Type":"ContainerStarted","Data":"3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae"} Jul 30 21:01:40.993255 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.993127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" event={"ID":"2451f3bb-f5a1-4211-82c1-36679047eb56","Type":"ContainerStarted","Data":"58c061f4e323b7e359bbd85fff5f19d77884b2699610ea5b8a6705e1a53ab5f3"} Jul 30 21:01:40.993255 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:40.993188 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:41.032558 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:41.032504 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" podStartSLOduration=1.03248835 podStartE2EDuration="1.03248835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:01:41.031146934 +0000 UTC m=+2439.496285283" watchObservedRunningTime="2026-07-30 21:01:41.03248835 +0000 UTC m=+2439.497626697" Jul 30 21:01:41.071541 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:41.071483 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" podStartSLOduration=1.071466028 podStartE2EDuration="1.071466028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:01:41.071184764 +0000 UTC m=+2439.536323112" watchObservedRunningTime="2026-07-30 21:01:41.071466028 +0000 UTC m=+2439.536604432" Jul 30 21:01:52.001598 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:52.001563 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:01:52.002125 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:01:52.001667 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:02:02.398353 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:02.398320 2575 scope.go:117] "RemoveContainer" containerID="ff3bd624d098658df345254f3f399853a99c270c10173ac138919fd678dde054" Jul 30 21:02:08.089358 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.089322 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6"] Jul 30 21:02:08.094165 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.094145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:08.096603 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.096581 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-12501c\"" Jul 30 21:02:08.102321 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.101973 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6"] Jul 30 21:02:08.107036 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.107011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8042799-88d7-4c7f-8ee7-75221e9a109c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6\" (UID: \"a8042799-88d7-4c7f-8ee7-75221e9a109c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:08.107163 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.107089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72k96\" (UniqueName: \"kubernetes.io/projected/a8042799-88d7-4c7f-8ee7-75221e9a109c-kube-api-access-72k96\") pod \"payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6\" (UID: \"a8042799-88d7-4c7f-8ee7-75221e9a109c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:08.123467 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.123441 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j"] Jul 30 21:02:08.127704 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.127686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:08.130588 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.130570 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-12501c-dockercfg-t82d9\"" Jul 30 21:02:08.138790 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.138766 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j"] Jul 30 21:02:08.208614 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.208530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8042799-88d7-4c7f-8ee7-75221e9a109c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6\" (UID: \"a8042799-88d7-4c7f-8ee7-75221e9a109c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:08.208814 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.208717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72k96\" (UniqueName: \"kubernetes.io/projected/a8042799-88d7-4c7f-8ee7-75221e9a109c-kube-api-access-72k96\") pod \"payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6\" (UID: \"a8042799-88d7-4c7f-8ee7-75221e9a109c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:08.208906 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.208808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9da13ce-95bc-4b0f-96ec-a3980933f88b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j\" (UID: \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:08.208976 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.208901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zn6s\" (UniqueName: \"kubernetes.io/projected/a9da13ce-95bc-4b0f-96ec-a3980933f88b-kube-api-access-8zn6s\") pod \"payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j\" (UID: \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:08.209576 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.209550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8042799-88d7-4c7f-8ee7-75221e9a109c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6\" (UID: \"a8042799-88d7-4c7f-8ee7-75221e9a109c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:08.218133 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.218109 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72k96\" (UniqueName: \"kubernetes.io/projected/a8042799-88d7-4c7f-8ee7-75221e9a109c-kube-api-access-72k96\") pod \"payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6\" (UID: \"a8042799-88d7-4c7f-8ee7-75221e9a109c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:08.310232 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.310196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9da13ce-95bc-4b0f-96ec-a3980933f88b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j\" (UID: \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:08.310232 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.310238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zn6s\" (UniqueName: \"kubernetes.io/projected/a9da13ce-95bc-4b0f-96ec-a3980933f88b-kube-api-access-8zn6s\") pod \"payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j\" (UID: \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:08.311041 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.311013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9da13ce-95bc-4b0f-96ec-a3980933f88b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j\" (UID: \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:08.322072 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.322035 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zn6s\" (UniqueName: \"kubernetes.io/projected/a9da13ce-95bc-4b0f-96ec-a3980933f88b-kube-api-access-8zn6s\") pod \"payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j\" (UID: \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:08.407297 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.407202 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:08.439546 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.439515 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:08.575368 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.575290 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6"] Jul 30 21:02:08.602315 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:08.602286 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j"] Jul 30 21:02:08.604149 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:02:08.604124 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9da13ce_95bc_4b0f_96ec_a3980933f88b.slice/crio-40e245a400108bf6b94d1add8abc8a1732d0518abaf8d70646603e3135d07e33 WatchSource:0}: Error finding container 40e245a400108bf6b94d1add8abc8a1732d0518abaf8d70646603e3135d07e33: Status 404 returned error can't find the container with id 40e245a400108bf6b94d1add8abc8a1732d0518abaf8d70646603e3135d07e33 Jul 30 21:02:09.156224 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:09.156179 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" event={"ID":"a8042799-88d7-4c7f-8ee7-75221e9a109c","Type":"ContainerStarted","Data":"4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971"} Jul 30 21:02:09.156697 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:09.156371 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:02:09.156697 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:09.156403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" event={"ID":"a8042799-88d7-4c7f-8ee7-75221e9a109c","Type":"ContainerStarted","Data":"63d6654052be6fb464a1284a42e99d802bebbaac404c8f26533de27e9c043c8f"} Jul 30 21:02:09.157751 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:09.157726 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" event={"ID":"a9da13ce-95bc-4b0f-96ec-a3980933f88b","Type":"ContainerStarted","Data":"722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a"} Jul 30 21:02:09.157856 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:09.157775 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" event={"ID":"a9da13ce-95bc-4b0f-96ec-a3980933f88b","Type":"ContainerStarted","Data":"40e245a400108bf6b94d1add8abc8a1732d0518abaf8d70646603e3135d07e33"} Jul 30 21:02:09.157856 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:09.157797 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:09.172781 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:09.172728 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" podStartSLOduration=1.172712341 podStartE2EDuration="1.172712341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:02:09.170874908 +0000 UTC m=+2467.636013267" watchObservedRunningTime="2026-07-30 21:02:09.172712341 +0000 UTC m=+2467.637850686" Jul 30 21:02:09.187489 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:09.187429 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" podStartSLOduration=1.187411236 podStartE2EDuration="1.187411236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:02:09.185991024 +0000 UTC m=+2467.651129374" watchObservedRunningTime="2026-07-30 21:02:09.187411236 +0000 UTC m=+2467.652549584" Jul 30 21:02:20.166044 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:20.166011 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:02:20.166494 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:02:20.166109 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:03:36.788241 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:36.788208 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg"] Jul 30 21:03:36.791959 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:36.791935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" Jul 30 21:03:36.794817 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:36.794791 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:03:36.799810 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:36.799781 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg"] Jul 30 21:03:36.933018 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:36.932985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c45b21-95b9-46fd-9dfe-2036d3ee296a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg\" (UID: \"d9c45b21-95b9-46fd-9dfe-2036d3ee296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" Jul 30 21:03:37.033869 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:37.033831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c45b21-95b9-46fd-9dfe-2036d3ee296a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg\" (UID: \"d9c45b21-95b9-46fd-9dfe-2036d3ee296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" Jul 30 21:03:37.034528 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:37.034503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c45b21-95b9-46fd-9dfe-2036d3ee296a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg\" (UID: \"d9c45b21-95b9-46fd-9dfe-2036d3ee296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" Jul 30 21:03:37.104579 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:37.104472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" Jul 30 21:03:37.253167 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:37.253133 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg"] Jul 30 21:03:37.254760 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:03:37.254732 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c45b21_95b9_46fd_9dfe_2036d3ee296a.slice/crio-75016fa5b8fe881c8c0849d7b64c076fe477dc21d0842d0889326cb20fdc17d4 WatchSource:0}: Error finding container 75016fa5b8fe881c8c0849d7b64c076fe477dc21d0842d0889326cb20fdc17d4: Status 404 returned error can't find the container with id 75016fa5b8fe881c8c0849d7b64c076fe477dc21d0842d0889326cb20fdc17d4 Jul 30 21:03:37.662116 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:37.662079 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" event={"ID":"d9c45b21-95b9-46fd-9dfe-2036d3ee296a","Type":"ContainerStarted","Data":"da995bf7b8dfdc980178335ba6bb95b3b34ff397e1efdf99ce3033ceb1595390"} Jul 30 21:03:37.662330 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:37.662153 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" event={"ID":"d9c45b21-95b9-46fd-9dfe-2036d3ee296a","Type":"ContainerStarted","Data":"75016fa5b8fe881c8c0849d7b64c076fe477dc21d0842d0889326cb20fdc17d4"} Jul 30 21:03:37.679990 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:37.679936 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" podStartSLOduration=1.6799191530000002 podStartE2EDuration="1.679919153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:03:37.679169139 +0000 UTC m=+2556.144307488" watchObservedRunningTime="2026-07-30 21:03:37.679919153 +0000 UTC m=+2556.145057500" Jul 30 21:03:38.669389 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:38.669353 2575 generic.go:356] "Generic (PLEG): container finished" podID="d9c45b21-95b9-46fd-9dfe-2036d3ee296a" containerID="da995bf7b8dfdc980178335ba6bb95b3b34ff397e1efdf99ce3033ceb1595390" exitCode=0 Jul 30 21:03:38.669802 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:38.669425 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" event={"ID":"d9c45b21-95b9-46fd-9dfe-2036d3ee296a","Type":"ContainerDied","Data":"da995bf7b8dfdc980178335ba6bb95b3b34ff397e1efdf99ce3033ceb1595390"} Jul 30 21:03:39.814817 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:39.814791 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" Jul 30 21:03:39.856986 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:39.856939 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c45b21-95b9-46fd-9dfe-2036d3ee296a-maas-api-service-ca\") pod \"d9c45b21-95b9-46fd-9dfe-2036d3ee296a\" (UID: \"d9c45b21-95b9-46fd-9dfe-2036d3ee296a\") " Jul 30 21:03:39.857316 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:39.857294 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c45b21-95b9-46fd-9dfe-2036d3ee296a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9c45b21-95b9-46fd-9dfe-2036d3ee296a" (UID: "d9c45b21-95b9-46fd-9dfe-2036d3ee296a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:03:39.957587 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:39.957555 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c45b21-95b9-46fd-9dfe-2036d3ee296a-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:03:40.688246 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:40.688216 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" Jul 30 21:03:40.688246 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:40.688239 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg" event={"ID":"d9c45b21-95b9-46fd-9dfe-2036d3ee296a","Type":"ContainerDied","Data":"75016fa5b8fe881c8c0849d7b64c076fe477dc21d0842d0889326cb20fdc17d4"} Jul 30 21:03:40.688459 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:40.688281 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75016fa5b8fe881c8c0849d7b64c076fe477dc21d0842d0889326cb20fdc17d4" Jul 30 21:03:47.266633 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.266594 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5"] Jul 30 21:03:47.267090 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.266875 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" podUID="2451f3bb-f5a1-4211-82c1-36679047eb56" containerName="payload-processing" containerID="cri-o://3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae" gracePeriod=30 Jul 30 21:03:47.272831 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.272800 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg"] Jul 30 21:03:47.273345 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.273302 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" podUID="33a08a0e-8f23-47a3-a71b-e09d01fe503d" containerName="payload-pre-processing" containerID="cri-o://5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f" gracePeriod=30 Jul 30 21:03:47.425860 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.424596 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg"] Jul 30 21:03:47.428729 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.428698 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72f733-bvxkg"] Jul 30 21:03:47.562702 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.562674 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:03:47.565845 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.565818 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:03:47.628550 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.628508 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a08a0e-8f23-47a3-a71b-e09d01fe503d-plugins-config-volume\") pod \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\" (UID: \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\") " Jul 30 21:03:47.628760 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.628563 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv6k4\" (UniqueName: \"kubernetes.io/projected/2451f3bb-f5a1-4211-82c1-36679047eb56-kube-api-access-jv6k4\") pod \"2451f3bb-f5a1-4211-82c1-36679047eb56\" (UID: \"2451f3bb-f5a1-4211-82c1-36679047eb56\") " Jul 30 21:03:47.628760 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.628597 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcs8g\" (UniqueName: \"kubernetes.io/projected/33a08a0e-8f23-47a3-a71b-e09d01fe503d-kube-api-access-tcs8g\") pod \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\" (UID: \"33a08a0e-8f23-47a3-a71b-e09d01fe503d\") " Jul 30 21:03:47.628760 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.628698 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451f3bb-f5a1-4211-82c1-36679047eb56-plugins-config-volume\") pod \"2451f3bb-f5a1-4211-82c1-36679047eb56\" (UID: \"2451f3bb-f5a1-4211-82c1-36679047eb56\") " Jul 30 21:03:47.629208 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.629160 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a08a0e-8f23-47a3-a71b-e09d01fe503d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33a08a0e-8f23-47a3-a71b-e09d01fe503d" (UID: "33a08a0e-8f23-47a3-a71b-e09d01fe503d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:03:47.629339 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.629201 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2451f3bb-f5a1-4211-82c1-36679047eb56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2451f3bb-f5a1-4211-82c1-36679047eb56" (UID: "2451f3bb-f5a1-4211-82c1-36679047eb56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:03:47.630906 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.630879 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a08a0e-8f23-47a3-a71b-e09d01fe503d-kube-api-access-tcs8g" (OuterVolumeSpecName: "kube-api-access-tcs8g") pod "33a08a0e-8f23-47a3-a71b-e09d01fe503d" (UID: "33a08a0e-8f23-47a3-a71b-e09d01fe503d"). InnerVolumeSpecName "kube-api-access-tcs8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:03:47.631000 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.630907 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2451f3bb-f5a1-4211-82c1-36679047eb56-kube-api-access-jv6k4" (OuterVolumeSpecName: "kube-api-access-jv6k4") pod "2451f3bb-f5a1-4211-82c1-36679047eb56" (UID: "2451f3bb-f5a1-4211-82c1-36679047eb56"). InnerVolumeSpecName "kube-api-access-jv6k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:03:47.728923 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.728885 2575 generic.go:356] "Generic (PLEG): container finished" podID="33a08a0e-8f23-47a3-a71b-e09d01fe503d" containerID="5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f" exitCode=0 Jul 30 21:03:47.729120 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.728928 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" Jul 30 21:03:47.729120 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.728969 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" event={"ID":"33a08a0e-8f23-47a3-a71b-e09d01fe503d","Type":"ContainerDied","Data":"5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f"} Jul 30 21:03:47.729120 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.729025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg" event={"ID":"33a08a0e-8f23-47a3-a71b-e09d01fe503d","Type":"ContainerDied","Data":"e82cf18701017b7d50b2a1ac33e8e768cf8b215e479b83a577cbce87666211dc"} Jul 30 21:03:47.729120 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.729048 2575 scope.go:117] "RemoveContainer" containerID="5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f" Jul 30 21:03:47.729530 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.729506 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a08a0e-8f23-47a3-a71b-e09d01fe503d-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:03:47.729530 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.729530 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv6k4\" (UniqueName: \"kubernetes.io/projected/2451f3bb-f5a1-4211-82c1-36679047eb56-kube-api-access-jv6k4\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:03:47.729676 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.729541 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcs8g\" (UniqueName: \"kubernetes.io/projected/33a08a0e-8f23-47a3-a71b-e09d01fe503d-kube-api-access-tcs8g\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:03:47.729676 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.729553 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451f3bb-f5a1-4211-82c1-36679047eb56-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:03:47.730588 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.730560 2575 generic.go:356] "Generic (PLEG): container finished" podID="2451f3bb-f5a1-4211-82c1-36679047eb56" containerID="3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae" exitCode=0 Jul 30 21:03:47.730697 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.730606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" event={"ID":"2451f3bb-f5a1-4211-82c1-36679047eb56","Type":"ContainerDied","Data":"3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae"} Jul 30 21:03:47.730697 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.730615 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" Jul 30 21:03:47.730697 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.730656 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5" event={"ID":"2451f3bb-f5a1-4211-82c1-36679047eb56","Type":"ContainerDied","Data":"58c061f4e323b7e359bbd85fff5f19d77884b2699610ea5b8a6705e1a53ab5f3"} Jul 30 21:03:47.741127 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.740873 2575 scope.go:117] "RemoveContainer" containerID="5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f" Jul 30 21:03:47.742506 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:03:47.742395 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f\": container with ID starting with 5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f not found: ID does not exist" containerID="5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f" Jul 30 21:03:47.742506 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.742438 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f"} err="failed to get container status \"5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f\": rpc error: code = NotFound desc = could not find container \"5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f\": container with ID starting with 5a4004e7b7b3e8fa4e316096e157b0811302367ebb93b113e0ca33add448078f not found: ID does not exist" Jul 30 21:03:47.742506 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.742463 2575 scope.go:117] "RemoveContainer" containerID="3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae" Jul 30 21:03:47.751340 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.751319 2575 scope.go:117] "RemoveContainer" containerID="3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae" Jul 30 21:03:47.751616 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:03:47.751600 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae\": container with ID starting with 3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae not found: ID does not exist" containerID="3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae" Jul 30 21:03:47.751666 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.751624 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae"} err="failed to get container status \"3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae\": rpc error: code = NotFound desc = could not find container \"3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae\": container with ID starting with 3081658dc18f9ce6f956d47684fee5209039d53df5b3daa9b3a756c5fce4b8ae not found: ID does not exist" Jul 30 21:03:47.757613 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.757586 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5"] Jul 30 21:03:47.759870 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.759849 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f733-7fcdd4d954-qrdc5"] Jul 30 21:03:47.769906 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.769881 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg"] Jul 30 21:03:47.775945 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:47.775918 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f733-6fdbc7b5b8-gcqsg"] Jul 30 21:03:48.013283 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:48.013241 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2451f3bb-f5a1-4211-82c1-36679047eb56" path="/var/lib/kubelet/pods/2451f3bb-f5a1-4211-82c1-36679047eb56/volumes" Jul 30 21:03:48.013787 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:48.013766 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a08a0e-8f23-47a3-a71b-e09d01fe503d" path="/var/lib/kubelet/pods/33a08a0e-8f23-47a3-a71b-e09d01fe503d/volumes" Jul 30 21:03:48.014166 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:48.014147 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c45b21-95b9-46fd-9dfe-2036d3ee296a" path="/var/lib/kubelet/pods/d9c45b21-95b9-46fd-9dfe-2036d3ee296a/volumes" Jul 30 21:03:54.308887 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.308848 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n"] Jul 30 21:03:54.309430 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309409 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2451f3bb-f5a1-4211-82c1-36679047eb56" containerName="payload-processing" Jul 30 21:03:54.309513 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309432 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2451f3bb-f5a1-4211-82c1-36679047eb56" containerName="payload-processing" Jul 30 21:03:54.309513 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309444 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c45b21-95b9-46fd-9dfe-2036d3ee296a" containerName="revoke-keys" Jul 30 21:03:54.309513 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309452 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c45b21-95b9-46fd-9dfe-2036d3ee296a" containerName="revoke-keys" Jul 30 21:03:54.309513 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309496 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a08a0e-8f23-47a3-a71b-e09d01fe503d" containerName="payload-pre-processing" Jul 30 21:03:54.309513 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309505 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a08a0e-8f23-47a3-a71b-e09d01fe503d" containerName="payload-pre-processing" Jul 30 21:03:54.309775 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309597 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2451f3bb-f5a1-4211-82c1-36679047eb56" containerName="payload-processing" Jul 30 21:03:54.309775 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309613 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c45b21-95b9-46fd-9dfe-2036d3ee296a" containerName="revoke-keys" Jul 30 21:03:54.309775 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.309625 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a08a0e-8f23-47a3-a71b-e09d01fe503d" containerName="payload-pre-processing" Jul 30 21:03:54.314157 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.314132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" Jul 30 21:03:54.317149 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.317106 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:03:54.320806 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.320218 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n"] Jul 30 21:03:54.392312 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.392275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb76980c-409c-4f52-92ab-bee3ead1d36b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n\" (UID: \"eb76980c-409c-4f52-92ab-bee3ead1d36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" Jul 30 21:03:54.493466 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.493425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb76980c-409c-4f52-92ab-bee3ead1d36b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n\" (UID: \"eb76980c-409c-4f52-92ab-bee3ead1d36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" Jul 30 21:03:54.494182 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.494162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb76980c-409c-4f52-92ab-bee3ead1d36b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n\" (UID: \"eb76980c-409c-4f52-92ab-bee3ead1d36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" Jul 30 21:03:54.629018 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.628936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" Jul 30 21:03:54.778619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:54.778583 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n"] Jul 30 21:03:54.779216 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:03:54.779188 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb76980c_409c_4f52_92ab_bee3ead1d36b.slice/crio-29c7868cfee8181bf645b1ff6d5d7fc4ae0d1a03f0f8e288184cd24c3a203172 WatchSource:0}: Error finding container 29c7868cfee8181bf645b1ff6d5d7fc4ae0d1a03f0f8e288184cd24c3a203172: Status 404 returned error can't find the container with id 29c7868cfee8181bf645b1ff6d5d7fc4ae0d1a03f0f8e288184cd24c3a203172 Jul 30 21:03:55.778199 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:55.778096 2575 generic.go:356] "Generic (PLEG): container finished" podID="eb76980c-409c-4f52-92ab-bee3ead1d36b" containerID="403def8a49492bd73b0042c4d719b6c8bcfdedd40b7ed0ea707115216e72655c" exitCode=0 Jul 30 21:03:55.778567 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:55.778185 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" event={"ID":"eb76980c-409c-4f52-92ab-bee3ead1d36b","Type":"ContainerDied","Data":"403def8a49492bd73b0042c4d719b6c8bcfdedd40b7ed0ea707115216e72655c"} Jul 30 21:03:55.778567 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:55.778250 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" event={"ID":"eb76980c-409c-4f52-92ab-bee3ead1d36b","Type":"ContainerStarted","Data":"29c7868cfee8181bf645b1ff6d5d7fc4ae0d1a03f0f8e288184cd24c3a203172"} Jul 30 21:03:56.923830 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:56.923802 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" Jul 30 21:03:57.016365 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:57.016325 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb76980c-409c-4f52-92ab-bee3ead1d36b-maas-api-service-ca\") pod \"eb76980c-409c-4f52-92ab-bee3ead1d36b\" (UID: \"eb76980c-409c-4f52-92ab-bee3ead1d36b\") " Jul 30 21:03:57.016720 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:57.016690 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb76980c-409c-4f52-92ab-bee3ead1d36b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb76980c-409c-4f52-92ab-bee3ead1d36b" (UID: "eb76980c-409c-4f52-92ab-bee3ead1d36b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:03:57.117761 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:57.117669 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb76980c-409c-4f52-92ab-bee3ead1d36b-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:03:57.790626 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:57.790597 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" Jul 30 21:03:57.790831 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:57.790636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n" event={"ID":"eb76980c-409c-4f52-92ab-bee3ead1d36b","Type":"ContainerDied","Data":"29c7868cfee8181bf645b1ff6d5d7fc4ae0d1a03f0f8e288184cd24c3a203172"} Jul 30 21:03:57.790831 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:03:57.790684 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29c7868cfee8181bf645b1ff6d5d7fc4ae0d1a03f0f8e288184cd24c3a203172" Jul 30 21:04:04.916857 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:04.916818 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j"] Jul 30 21:04:04.917304 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:04.917103 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" podUID="a9da13ce-95bc-4b0f-96ec-a3980933f88b" containerName="payload-processing" containerID="cri-o://722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a" gracePeriod=30 Jul 30 21:04:04.928612 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:04.928584 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6"] Jul 30 21:04:04.928881 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:04.928854 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" podUID="a8042799-88d7-4c7f-8ee7-75221e9a109c" containerName="payload-pre-processing" containerID="cri-o://4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971" gracePeriod=30 Jul 30 21:04:05.059993 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.059927 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n"] Jul 30 21:04:05.062768 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.062736 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12501c-d5x4n"] Jul 30 21:04:05.193845 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.193818 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:04:05.218462 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.218432 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:04:05.286309 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.286272 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zn6s\" (UniqueName: \"kubernetes.io/projected/a9da13ce-95bc-4b0f-96ec-a3980933f88b-kube-api-access-8zn6s\") pod \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\" (UID: \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\") " Jul 30 21:04:05.286489 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.286350 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8042799-88d7-4c7f-8ee7-75221e9a109c-plugins-config-volume\") pod \"a8042799-88d7-4c7f-8ee7-75221e9a109c\" (UID: \"a8042799-88d7-4c7f-8ee7-75221e9a109c\") " Jul 30 21:04:05.286489 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.286374 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72k96\" (UniqueName: \"kubernetes.io/projected/a8042799-88d7-4c7f-8ee7-75221e9a109c-kube-api-access-72k96\") pod \"a8042799-88d7-4c7f-8ee7-75221e9a109c\" (UID: \"a8042799-88d7-4c7f-8ee7-75221e9a109c\") " Jul 30 21:04:05.286489 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.286453 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9da13ce-95bc-4b0f-96ec-a3980933f88b-plugins-config-volume\") pod \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\" (UID: \"a9da13ce-95bc-4b0f-96ec-a3980933f88b\") " Jul 30 21:04:05.286803 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.286776 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8042799-88d7-4c7f-8ee7-75221e9a109c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8042799-88d7-4c7f-8ee7-75221e9a109c" (UID: "a8042799-88d7-4c7f-8ee7-75221e9a109c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:04:05.286924 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.286867 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9da13ce-95bc-4b0f-96ec-a3980933f88b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9da13ce-95bc-4b0f-96ec-a3980933f88b" (UID: "a9da13ce-95bc-4b0f-96ec-a3980933f88b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:04:05.288393 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.288370 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9da13ce-95bc-4b0f-96ec-a3980933f88b-kube-api-access-8zn6s" (OuterVolumeSpecName: "kube-api-access-8zn6s") pod "a9da13ce-95bc-4b0f-96ec-a3980933f88b" (UID: "a9da13ce-95bc-4b0f-96ec-a3980933f88b"). InnerVolumeSpecName "kube-api-access-8zn6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:04:05.288534 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.288518 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8042799-88d7-4c7f-8ee7-75221e9a109c-kube-api-access-72k96" (OuterVolumeSpecName: "kube-api-access-72k96") pod "a8042799-88d7-4c7f-8ee7-75221e9a109c" (UID: "a8042799-88d7-4c7f-8ee7-75221e9a109c"). InnerVolumeSpecName "kube-api-access-72k96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:04:05.387899 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.387856 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9da13ce-95bc-4b0f-96ec-a3980933f88b-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:04:05.387899 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.387898 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zn6s\" (UniqueName: \"kubernetes.io/projected/a9da13ce-95bc-4b0f-96ec-a3980933f88b-kube-api-access-8zn6s\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:04:05.388178 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.387914 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8042799-88d7-4c7f-8ee7-75221e9a109c-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:04:05.388178 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.387927 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72k96\" (UniqueName: \"kubernetes.io/projected/a8042799-88d7-4c7f-8ee7-75221e9a109c-kube-api-access-72k96\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:04:05.834625 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.834593 2575 generic.go:356] "Generic (PLEG): container finished" podID="a9da13ce-95bc-4b0f-96ec-a3980933f88b" containerID="722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a" exitCode=0 Jul 30 21:04:05.834822 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.834638 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" Jul 30 21:04:05.834822 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.834681 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" event={"ID":"a9da13ce-95bc-4b0f-96ec-a3980933f88b","Type":"ContainerDied","Data":"722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a"} Jul 30 21:04:05.834822 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.834738 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j" event={"ID":"a9da13ce-95bc-4b0f-96ec-a3980933f88b","Type":"ContainerDied","Data":"40e245a400108bf6b94d1add8abc8a1732d0518abaf8d70646603e3135d07e33"} Jul 30 21:04:05.834822 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.834769 2575 scope.go:117] "RemoveContainer" containerID="722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a" Jul 30 21:04:05.836208 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.836185 2575 generic.go:356] "Generic (PLEG): container finished" podID="a8042799-88d7-4c7f-8ee7-75221e9a109c" containerID="4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971" exitCode=0 Jul 30 21:04:05.836308 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.836231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" event={"ID":"a8042799-88d7-4c7f-8ee7-75221e9a109c","Type":"ContainerDied","Data":"4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971"} Jul 30 21:04:05.836308 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.836271 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" event={"ID":"a8042799-88d7-4c7f-8ee7-75221e9a109c","Type":"ContainerDied","Data":"63d6654052be6fb464a1284a42e99d802bebbaac404c8f26533de27e9c043c8f"} Jul 30 21:04:05.836308 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.836236 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6" Jul 30 21:04:05.846981 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.846938 2575 scope.go:117] "RemoveContainer" containerID="722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a" Jul 30 21:04:05.847686 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:04:05.847661 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a\": container with ID starting with 722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a not found: ID does not exist" containerID="722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a" Jul 30 21:04:05.847805 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.847691 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a"} err="failed to get container status \"722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a\": rpc error: code = NotFound desc = could not find container \"722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a\": container with ID starting with 722e84c3a9adba62830a3ea9984ec2b9c595a372151716f4f8eca58f27cb598a not found: ID does not exist" Jul 30 21:04:05.847805 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.847706 2575 scope.go:117] "RemoveContainer" containerID="4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971" Jul 30 21:04:05.858520 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.858500 2575 scope.go:117] "RemoveContainer" containerID="4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971" Jul 30 21:04:05.858813 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:04:05.858791 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971\": container with ID starting with 4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971 not found: ID does not exist" containerID="4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971" Jul 30 21:04:05.858885 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.858822 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971"} err="failed to get container status \"4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971\": rpc error: code = NotFound desc = could not find container \"4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971\": container with ID starting with 4447497b906592d1847d114a70b61a6952ff644921db05976b7607fb50f00971 not found: ID does not exist" Jul 30 21:04:05.871480 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.871456 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j"] Jul 30 21:04:05.877649 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.877625 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12501c-67bc9dcd5b-98q9j"] Jul 30 21:04:05.889346 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.889309 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6"] Jul 30 21:04:05.893293 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:05.893270 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12501c-86cf5c8856-xjms6"] Jul 30 21:04:06.013699 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:06.013648 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8042799-88d7-4c7f-8ee7-75221e9a109c" path="/var/lib/kubelet/pods/a8042799-88d7-4c7f-8ee7-75221e9a109c/volumes" Jul 30 21:04:06.014225 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:06.014201 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9da13ce-95bc-4b0f-96ec-a3980933f88b" path="/var/lib/kubelet/pods/a9da13ce-95bc-4b0f-96ec-a3980933f88b/volumes" Jul 30 21:04:06.014512 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:06.014499 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb76980c-409c-4f52-92ab-bee3ead1d36b" path="/var/lib/kubelet/pods/eb76980c-409c-4f52-92ab-bee3ead1d36b/volumes" Jul 30 21:04:16.867256 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867210 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b"] Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867585 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8042799-88d7-4c7f-8ee7-75221e9a109c" containerName="payload-pre-processing" Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867597 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8042799-88d7-4c7f-8ee7-75221e9a109c" containerName="payload-pre-processing" Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867612 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb76980c-409c-4f52-92ab-bee3ead1d36b" containerName="revoke-keys" Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867618 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb76980c-409c-4f52-92ab-bee3ead1d36b" containerName="revoke-keys" Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867631 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9da13ce-95bc-4b0f-96ec-a3980933f88b" containerName="payload-processing" Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867637 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9da13ce-95bc-4b0f-96ec-a3980933f88b" containerName="payload-processing" Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867691 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8042799-88d7-4c7f-8ee7-75221e9a109c" containerName="payload-pre-processing" Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867700 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9da13ce-95bc-4b0f-96ec-a3980933f88b" containerName="payload-processing" Jul 30 21:04:16.867832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.867706 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb76980c-409c-4f52-92ab-bee3ead1d36b" containerName="revoke-keys" Jul 30 21:04:16.870909 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.870889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:16.873680 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.873652 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-99c4a7\"" Jul 30 21:04:16.881367 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.881329 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b"] Jul 30 21:04:16.890306 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.890265 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r"] Jul 30 21:04:16.894136 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.894101 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:16.897081 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.897041 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-99c4a7-dockercfg-jt8d6\"" Jul 30 21:04:16.907736 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.907692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r"] Jul 30 21:04:16.995917 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.995884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mxq\" (UniqueName: \"kubernetes.io/projected/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-kube-api-access-k8mxq\") pod \"payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b\" (UID: \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:16.996128 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.995950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b\" (UID: \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:16.996128 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.995995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pfcd\" (UniqueName: \"kubernetes.io/projected/a8bc981f-dbc7-468b-8768-916fb22853d6-kube-api-access-8pfcd\") pod \"payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r\" (UID: \"a8bc981f-dbc7-468b-8768-916fb22853d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:16.996128 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:16.996094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bc981f-dbc7-468b-8768-916fb22853d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r\" (UID: \"a8bc981f-dbc7-468b-8768-916fb22853d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:17.097142 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.097097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b\" (UID: \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:17.097376 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.097162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pfcd\" (UniqueName: \"kubernetes.io/projected/a8bc981f-dbc7-468b-8768-916fb22853d6-kube-api-access-8pfcd\") pod \"payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r\" (UID: \"a8bc981f-dbc7-468b-8768-916fb22853d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:17.097376 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.097208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bc981f-dbc7-468b-8768-916fb22853d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r\" (UID: \"a8bc981f-dbc7-468b-8768-916fb22853d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:17.097494 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.097381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mxq\" (UniqueName: \"kubernetes.io/projected/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-kube-api-access-k8mxq\") pod \"payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b\" (UID: \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:17.098141 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.098114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bc981f-dbc7-468b-8768-916fb22853d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r\" (UID: \"a8bc981f-dbc7-468b-8768-916fb22853d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:17.098260 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.098243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b\" (UID: \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:17.124000 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.123920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mxq\" (UniqueName: \"kubernetes.io/projected/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-kube-api-access-k8mxq\") pod \"payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b\" (UID: \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:17.145147 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.145114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pfcd\" (UniqueName: \"kubernetes.io/projected/a8bc981f-dbc7-468b-8768-916fb22853d6-kube-api-access-8pfcd\") pod \"payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r\" (UID: \"a8bc981f-dbc7-468b-8768-916fb22853d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:17.183900 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.183864 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:17.209444 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.209408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:17.350949 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.350848 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b"] Jul 30 21:04:17.381513 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.381484 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r"] Jul 30 21:04:17.388314 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:04:17.388288 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bc981f_dbc7_468b_8768_916fb22853d6.slice/crio-e35bc50f3d7e9461e994d1a07b83042d1f5e87f0a248a2f474d12c7ec8b165e8 WatchSource:0}: Error finding container e35bc50f3d7e9461e994d1a07b83042d1f5e87f0a248a2f474d12c7ec8b165e8: Status 404 returned error can't find the container with id e35bc50f3d7e9461e994d1a07b83042d1f5e87f0a248a2f474d12c7ec8b165e8 Jul 30 21:04:17.906650 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.906614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" event={"ID":"a8bc981f-dbc7-468b-8768-916fb22853d6","Type":"ContainerStarted","Data":"17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853"} Jul 30 21:04:17.907147 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.906672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" event={"ID":"a8bc981f-dbc7-468b-8768-916fb22853d6","Type":"ContainerStarted","Data":"e35bc50f3d7e9461e994d1a07b83042d1f5e87f0a248a2f474d12c7ec8b165e8"} Jul 30 21:04:17.907147 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.906731 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:17.908134 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.908113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" event={"ID":"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd","Type":"ContainerStarted","Data":"83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c"} Jul 30 21:04:17.908255 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.908158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" event={"ID":"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd","Type":"ContainerStarted","Data":"047678afea0d1d2bb8c868b310a866b00f8aee32e2ce12a5bf9375ad5d200841"} Jul 30 21:04:17.908255 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.908199 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:17.927090 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.924231 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" podStartSLOduration=1.924200979 podStartE2EDuration="1.924200979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:17.921491937 +0000 UTC m=+2596.386630297" watchObservedRunningTime="2026-07-30 21:04:17.924200979 +0000 UTC m=+2596.389339328" Jul 30 21:04:17.942336 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:17.942282 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" podStartSLOduration=1.942264355 podStartE2EDuration="1.942264355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:17.941312523 +0000 UTC m=+2596.406450872" watchObservedRunningTime="2026-07-30 21:04:17.942264355 +0000 UTC m=+2596.407402701" Jul 30 21:04:28.916237 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:28.916147 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:04:28.916689 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:28.916359 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:04:46.014130 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.014092 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b"] Jul 30 21:04:46.020141 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.019110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:46.024661 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.024633 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6e507a\"" Jul 30 21:04:46.025134 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.024693 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6e507a-dockercfg-z9trz\"" Jul 30 21:04:46.026113 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.026089 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b"] Jul 30 21:04:46.054232 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.054193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b\" (UID: \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:46.054456 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.054410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkft\" (UniqueName: \"kubernetes.io/projected/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-kube-api-access-mdkft\") pod \"payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b\" (UID: \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:46.060123 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.060096 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx"] Jul 30 21:04:46.063557 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.063539 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:46.074452 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.074427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx"] Jul 30 21:04:46.155717 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.155675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/09f483bf-4828-4a49-ab1a-ef46e7024a73-kube-api-access-cnnjv\") pod \"payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx\" (UID: \"09f483bf-4828-4a49-ab1a-ef46e7024a73\") " pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:46.155888 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.155810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkft\" (UniqueName: \"kubernetes.io/projected/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-kube-api-access-mdkft\") pod \"payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b\" (UID: \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:46.155975 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.155888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b\" (UID: \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:46.155975 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.155915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f483bf-4828-4a49-ab1a-ef46e7024a73-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx\" (UID: \"09f483bf-4828-4a49-ab1a-ef46e7024a73\") " pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:46.156631 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.156610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b\" (UID: \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:46.164324 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.164291 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkft\" (UniqueName: \"kubernetes.io/projected/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-kube-api-access-mdkft\") pod \"payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b\" (UID: \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:46.257122 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.257081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f483bf-4828-4a49-ab1a-ef46e7024a73-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx\" (UID: \"09f483bf-4828-4a49-ab1a-ef46e7024a73\") " pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:46.257345 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.257167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/09f483bf-4828-4a49-ab1a-ef46e7024a73-kube-api-access-cnnjv\") pod \"payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx\" (UID: \"09f483bf-4828-4a49-ab1a-ef46e7024a73\") " pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:46.257873 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.257846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f483bf-4828-4a49-ab1a-ef46e7024a73-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx\" (UID: \"09f483bf-4828-4a49-ab1a-ef46e7024a73\") " pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:46.267909 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.267834 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/09f483bf-4828-4a49-ab1a-ef46e7024a73-kube-api-access-cnnjv\") pod \"payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx\" (UID: \"09f483bf-4828-4a49-ab1a-ef46e7024a73\") " pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:46.334878 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.334831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:46.374516 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.374483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:46.493172 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.493132 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b"] Jul 30 21:04:46.496355 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:04:46.496315 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8bddab_d42e_4a66_8f55_7dea2eea56fc.slice/crio-a2e3c9b835da074737246e5ccc8e58d9f860a8b79e427958df9fd893b4fdbd71 WatchSource:0}: Error finding container a2e3c9b835da074737246e5ccc8e58d9f860a8b79e427958df9fd893b4fdbd71: Status 404 returned error can't find the container with id a2e3c9b835da074737246e5ccc8e58d9f860a8b79e427958df9fd893b4fdbd71 Jul 30 21:04:46.534968 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:46.534939 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx"] Jul 30 21:04:46.536389 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:04:46.536361 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f483bf_4828_4a49_ab1a_ef46e7024a73.slice/crio-fc16c4c03b751233ccc4637b53f118b29868524924576370cab2cb6a861caebb WatchSource:0}: Error finding container fc16c4c03b751233ccc4637b53f118b29868524924576370cab2cb6a861caebb: Status 404 returned error can't find the container with id fc16c4c03b751233ccc4637b53f118b29868524924576370cab2cb6a861caebb Jul 30 21:04:47.071896 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:47.071844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" event={"ID":"09f483bf-4828-4a49-ab1a-ef46e7024a73","Type":"ContainerStarted","Data":"75e4bf769a862f767ed85e6b843fca02c9a9aad8277f85c6b3b47ce4b0e5c46b"} Jul 30 21:04:47.072392 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:47.071917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" event={"ID":"09f483bf-4828-4a49-ab1a-ef46e7024a73","Type":"ContainerStarted","Data":"fc16c4c03b751233ccc4637b53f118b29868524924576370cab2cb6a861caebb"} Jul 30 21:04:47.072392 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:47.071945 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:47.073594 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:47.073564 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" event={"ID":"bc8bddab-d42e-4a66-8f55-7dea2eea56fc","Type":"ContainerStarted","Data":"9ebc5badebe3771ab363d1a947ea200ba0285054f1e3f4d1d4b2d390f5abcc43"} Jul 30 21:04:47.073705 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:47.073624 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:04:47.073705 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:47.073649 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" event={"ID":"bc8bddab-d42e-4a66-8f55-7dea2eea56fc","Type":"ContainerStarted","Data":"a2e3c9b835da074737246e5ccc8e58d9f860a8b79e427958df9fd893b4fdbd71"} Jul 30 21:04:47.088238 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:47.088179 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" podStartSLOduration=1.0881577 podStartE2EDuration="1.0881577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:47.087280748 +0000 UTC m=+2625.552419097" watchObservedRunningTime="2026-07-30 21:04:47.0881577 +0000 UTC m=+2625.553296050" Jul 30 21:04:47.105962 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:47.105904 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" podStartSLOduration=2.105884919 podStartE2EDuration="2.105884919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:47.104025697 +0000 UTC m=+2625.569164046" watchObservedRunningTime="2026-07-30 21:04:47.105884919 +0000 UTC m=+2625.571023267" Jul 30 21:04:58.081765 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:58.081726 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:04:58.082353 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:04:58.081829 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:05:12.235865 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.235824 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht"] Jul 30 21:05:12.239290 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.239265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" Jul 30 21:05:12.241829 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.241802 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:05:12.247116 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.247087 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht"] Jul 30 21:05:12.299378 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.299339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f158a9e-919e-4bf2-b3bd-c8bce108533e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht\" (UID: \"9f158a9e-919e-4bf2-b3bd-c8bce108533e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" Jul 30 21:05:12.400739 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.400701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f158a9e-919e-4bf2-b3bd-c8bce108533e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht\" (UID: \"9f158a9e-919e-4bf2-b3bd-c8bce108533e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" Jul 30 21:05:12.401407 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.401382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f158a9e-919e-4bf2-b3bd-c8bce108533e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht\" (UID: \"9f158a9e-919e-4bf2-b3bd-c8bce108533e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" Jul 30 21:05:12.552632 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.552537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" Jul 30 21:05:12.698181 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:12.698123 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht"] Jul 30 21:05:12.699839 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:05:12.699812 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f158a9e_919e_4bf2_b3bd_c8bce108533e.slice/crio-7292c2a6b0d2cc1eebfe00600c60ff0cbf90014fb23011f7eff99d32af1bd4c5 WatchSource:0}: Error finding container 7292c2a6b0d2cc1eebfe00600c60ff0cbf90014fb23011f7eff99d32af1bd4c5: Status 404 returned error can't find the container with id 7292c2a6b0d2cc1eebfe00600c60ff0cbf90014fb23011f7eff99d32af1bd4c5 Jul 30 21:05:13.228821 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:13.228774 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" event={"ID":"9f158a9e-919e-4bf2-b3bd-c8bce108533e","Type":"ContainerStarted","Data":"d95763c3baf7c0391ac4b1ff28f4a1f86109df77b680abfcfb88ff7ddfcd8942"} Jul 30 21:05:13.229036 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:13.228851 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" event={"ID":"9f158a9e-919e-4bf2-b3bd-c8bce108533e","Type":"ContainerStarted","Data":"7292c2a6b0d2cc1eebfe00600c60ff0cbf90014fb23011f7eff99d32af1bd4c5"} Jul 30 21:05:13.249276 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:13.249216 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" podStartSLOduration=1.249200597 podStartE2EDuration="1.249200597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:05:13.247693986 +0000 UTC m=+2651.712832334" watchObservedRunningTime="2026-07-30 21:05:13.249200597 +0000 UTC m=+2651.714338945" Jul 30 21:05:14.239196 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:14.239158 2575 generic.go:356] "Generic (PLEG): container finished" podID="9f158a9e-919e-4bf2-b3bd-c8bce108533e" containerID="d95763c3baf7c0391ac4b1ff28f4a1f86109df77b680abfcfb88ff7ddfcd8942" exitCode=0 Jul 30 21:05:14.239383 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:14.239238 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" event={"ID":"9f158a9e-919e-4bf2-b3bd-c8bce108533e","Type":"ContainerDied","Data":"d95763c3baf7c0391ac4b1ff28f4a1f86109df77b680abfcfb88ff7ddfcd8942"} Jul 30 21:05:15.386035 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:15.386005 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" Jul 30 21:05:15.431327 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:15.431288 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f158a9e-919e-4bf2-b3bd-c8bce108533e-maas-api-service-ca\") pod \"9f158a9e-919e-4bf2-b3bd-c8bce108533e\" (UID: \"9f158a9e-919e-4bf2-b3bd-c8bce108533e\") " Jul 30 21:05:15.431681 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:15.431653 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f158a9e-919e-4bf2-b3bd-c8bce108533e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f158a9e-919e-4bf2-b3bd-c8bce108533e" (UID: "9f158a9e-919e-4bf2-b3bd-c8bce108533e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:05:15.532311 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:15.532215 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f158a9e-919e-4bf2-b3bd-c8bce108533e-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:16.252360 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:16.252327 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" Jul 30 21:05:16.252550 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:16.252367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht" event={"ID":"9f158a9e-919e-4bf2-b3bd-c8bce108533e","Type":"ContainerDied","Data":"7292c2a6b0d2cc1eebfe00600c60ff0cbf90014fb23011f7eff99d32af1bd4c5"} Jul 30 21:05:16.252550 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:16.252424 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7292c2a6b0d2cc1eebfe00600c60ff0cbf90014fb23011f7eff99d32af1bd4c5" Jul 30 21:05:27.891357 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:27.891317 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r"] Jul 30 21:05:27.891841 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:27.891553 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" podUID="a8bc981f-dbc7-468b-8768-916fb22853d6" containerName="payload-processing" containerID="cri-o://17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853" gracePeriod=30 Jul 30 21:05:27.900269 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:27.900076 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b"] Jul 30 21:05:27.900426 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:27.900367 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" podUID="d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd" containerName="payload-pre-processing" containerID="cri-o://83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c" gracePeriod=30 Jul 30 21:05:28.045295 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.045252 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht"] Jul 30 21:05:28.047754 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.047699 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-99c4a7-gfxht"] Jul 30 21:05:28.186876 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.186850 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:05:28.189584 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.189558 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:05:28.237236 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.237202 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8mxq\" (UniqueName: \"kubernetes.io/projected/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-kube-api-access-k8mxq\") pod \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\" (UID: \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\") " Jul 30 21:05:28.237446 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.237257 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-plugins-config-volume\") pod \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\" (UID: \"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd\") " Jul 30 21:05:28.237446 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.237285 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bc981f-dbc7-468b-8768-916fb22853d6-plugins-config-volume\") pod \"a8bc981f-dbc7-468b-8768-916fb22853d6\" (UID: \"a8bc981f-dbc7-468b-8768-916fb22853d6\") " Jul 30 21:05:28.237446 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.237349 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pfcd\" (UniqueName: \"kubernetes.io/projected/a8bc981f-dbc7-468b-8768-916fb22853d6-kube-api-access-8pfcd\") pod \"a8bc981f-dbc7-468b-8768-916fb22853d6\" (UID: \"a8bc981f-dbc7-468b-8768-916fb22853d6\") " Jul 30 21:05:28.237694 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.237657 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd" (UID: "d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:05:28.237781 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.237751 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bc981f-dbc7-468b-8768-916fb22853d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8bc981f-dbc7-468b-8768-916fb22853d6" (UID: "a8bc981f-dbc7-468b-8768-916fb22853d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:05:28.239534 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.239504 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bc981f-dbc7-468b-8768-916fb22853d6-kube-api-access-8pfcd" (OuterVolumeSpecName: "kube-api-access-8pfcd") pod "a8bc981f-dbc7-468b-8768-916fb22853d6" (UID: "a8bc981f-dbc7-468b-8768-916fb22853d6"). InnerVolumeSpecName "kube-api-access-8pfcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:05:28.239631 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.239547 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-kube-api-access-k8mxq" (OuterVolumeSpecName: "kube-api-access-k8mxq") pod "d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd" (UID: "d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd"). InnerVolumeSpecName "kube-api-access-k8mxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:05:28.322991 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.322959 2575 generic.go:356] "Generic (PLEG): container finished" podID="a8bc981f-dbc7-468b-8768-916fb22853d6" containerID="17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853" exitCode=0 Jul 30 21:05:28.323188 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.323016 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" Jul 30 21:05:28.323188 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.323038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" event={"ID":"a8bc981f-dbc7-468b-8768-916fb22853d6","Type":"ContainerDied","Data":"17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853"} Jul 30 21:05:28.323188 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.323127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r" event={"ID":"a8bc981f-dbc7-468b-8768-916fb22853d6","Type":"ContainerDied","Data":"e35bc50f3d7e9461e994d1a07b83042d1f5e87f0a248a2f474d12c7ec8b165e8"} Jul 30 21:05:28.323188 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.323162 2575 scope.go:117] "RemoveContainer" containerID="17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853" Jul 30 21:05:28.324458 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.324438 2575 generic.go:356] "Generic (PLEG): container finished" podID="d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd" containerID="83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c" exitCode=0 Jul 30 21:05:28.324543 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.324509 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" Jul 30 21:05:28.324543 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.324522 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" event={"ID":"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd","Type":"ContainerDied","Data":"83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c"} Jul 30 21:05:28.324656 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.324576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b" event={"ID":"d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd","Type":"ContainerDied","Data":"047678afea0d1d2bb8c868b310a866b00f8aee32e2ce12a5bf9375ad5d200841"} Jul 30 21:05:28.334929 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.334911 2575 scope.go:117] "RemoveContainer" containerID="17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853" Jul 30 21:05:28.335285 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:05:28.335262 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853\": container with ID starting with 17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853 not found: ID does not exist" containerID="17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853" Jul 30 21:05:28.335364 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.335295 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853"} err="failed to get container status \"17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853\": rpc error: code = NotFound desc = could not find container \"17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853\": container with ID starting with 17d34a1a3b85b72842ef40a3881d7c60e828318e32c6727ca8e8eb42e0959853 not found: ID does not exist" Jul 30 21:05:28.335364 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.335313 2575 scope.go:117] "RemoveContainer" containerID="83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c" Jul 30 21:05:28.338753 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.338728 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pfcd\" (UniqueName: \"kubernetes.io/projected/a8bc981f-dbc7-468b-8768-916fb22853d6-kube-api-access-8pfcd\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:28.338753 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.338753 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8mxq\" (UniqueName: \"kubernetes.io/projected/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-kube-api-access-k8mxq\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:28.338882 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.338766 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:28.338882 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.338776 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8bc981f-dbc7-468b-8768-916fb22853d6-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:28.344215 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.344186 2575 scope.go:117] "RemoveContainer" containerID="83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c" Jul 30 21:05:28.344620 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:05:28.344573 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c\": container with ID starting with 83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c not found: ID does not exist" containerID="83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c" Jul 30 21:05:28.344705 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.344609 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c"} err="failed to get container status \"83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c\": rpc error: code = NotFound desc = could not find container \"83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c\": container with ID starting with 83a53941c7a55e7e46ef8e81088e14bc295e958f003a8f9710d3833cd0dae34c not found: ID does not exist" Jul 30 21:05:28.347089 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.347066 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r"] Jul 30 21:05:28.351230 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.351209 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-99c4a7-cbd68c9bb-gdj2r"] Jul 30 21:05:28.362357 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.362317 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b"] Jul 30 21:05:28.366927 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:28.366900 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-99c4a7-7dbb48bd7b-bvx7b"] Jul 30 21:05:30.012824 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:30.012788 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f158a9e-919e-4bf2-b3bd-c8bce108533e" path="/var/lib/kubelet/pods/9f158a9e-919e-4bf2-b3bd-c8bce108533e/volumes" Jul 30 21:05:30.013295 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:30.013171 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bc981f-dbc7-468b-8768-916fb22853d6" path="/var/lib/kubelet/pods/a8bc981f-dbc7-468b-8768-916fb22853d6/volumes" Jul 30 21:05:30.013491 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:30.013477 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd" path="/var/lib/kubelet/pods/d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd/volumes" Jul 30 21:05:34.800000 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.799960 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7"] Jul 30 21:05:34.800531 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800434 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bc981f-dbc7-468b-8768-916fb22853d6" containerName="payload-processing" Jul 30 21:05:34.800531 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800448 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bc981f-dbc7-468b-8768-916fb22853d6" containerName="payload-processing" Jul 30 21:05:34.800531 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800460 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd" containerName="payload-pre-processing" Jul 30 21:05:34.800531 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800466 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd" containerName="payload-pre-processing" Jul 30 21:05:34.800531 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800475 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f158a9e-919e-4bf2-b3bd-c8bce108533e" containerName="revoke-keys" Jul 30 21:05:34.800531 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800480 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f158a9e-919e-4bf2-b3bd-c8bce108533e" containerName="revoke-keys" Jul 30 21:05:34.800531 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800535 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bc981f-dbc7-468b-8768-916fb22853d6" containerName="payload-processing" Jul 30 21:05:34.800901 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800544 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f158a9e-919e-4bf2-b3bd-c8bce108533e" containerName="revoke-keys" Jul 30 21:05:34.800901 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.800550 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62b104d-de8f-4e45-8cbc-1c0f14bd1bbd" containerName="payload-pre-processing" Jul 30 21:05:34.804971 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.804946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" Jul 30 21:05:34.807986 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.807963 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:05:34.810907 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.810881 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7"] Jul 30 21:05:34.894686 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.894650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b84222f-097c-479b-9704-a62df77a6bbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7\" (UID: \"0b84222f-097c-479b-9704-a62df77a6bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" Jul 30 21:05:34.995456 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.995417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b84222f-097c-479b-9704-a62df77a6bbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7\" (UID: \"0b84222f-097c-479b-9704-a62df77a6bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" Jul 30 21:05:34.996015 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:34.995995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b84222f-097c-479b-9704-a62df77a6bbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7\" (UID: \"0b84222f-097c-479b-9704-a62df77a6bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" Jul 30 21:05:35.118125 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:35.118034 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" Jul 30 21:05:35.262247 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:35.262165 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7"] Jul 30 21:05:35.264852 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:05:35.264825 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b84222f_097c_479b_9704_a62df77a6bbd.slice/crio-ff6cbd2793280acdd65b19f270f335fbfb124850077f96cf0d1ef825df64c799 WatchSource:0}: Error finding container ff6cbd2793280acdd65b19f270f335fbfb124850077f96cf0d1ef825df64c799: Status 404 returned error can't find the container with id ff6cbd2793280acdd65b19f270f335fbfb124850077f96cf0d1ef825df64c799 Jul 30 21:05:35.372925 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:35.372837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" event={"ID":"0b84222f-097c-479b-9704-a62df77a6bbd","Type":"ContainerStarted","Data":"ff6cbd2793280acdd65b19f270f335fbfb124850077f96cf0d1ef825df64c799"} Jul 30 21:05:36.382611 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:36.382507 2575 generic.go:356] "Generic (PLEG): container finished" podID="0b84222f-097c-479b-9704-a62df77a6bbd" containerID="d05e70482afee2dd050e12a5408d7fa083e3f79cdbe0857a0c39746eb3bdf520" exitCode=0 Jul 30 21:05:36.382611 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:36.382577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" event={"ID":"0b84222f-097c-479b-9704-a62df77a6bbd","Type":"ContainerDied","Data":"d05e70482afee2dd050e12a5408d7fa083e3f79cdbe0857a0c39746eb3bdf520"} Jul 30 21:05:37.521146 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:37.521117 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" Jul 30 21:05:37.621357 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:37.621320 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b84222f-097c-479b-9704-a62df77a6bbd-maas-api-service-ca\") pod \"0b84222f-097c-479b-9704-a62df77a6bbd\" (UID: \"0b84222f-097c-479b-9704-a62df77a6bbd\") " Jul 30 21:05:37.621738 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:37.621703 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b84222f-097c-479b-9704-a62df77a6bbd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b84222f-097c-479b-9704-a62df77a6bbd" (UID: "0b84222f-097c-479b-9704-a62df77a6bbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:05:37.722664 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:37.722626 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b84222f-097c-479b-9704-a62df77a6bbd-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:38.395131 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:38.395034 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" Jul 30 21:05:38.395288 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:38.395026 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7" event={"ID":"0b84222f-097c-479b-9704-a62df77a6bbd","Type":"ContainerDied","Data":"ff6cbd2793280acdd65b19f270f335fbfb124850077f96cf0d1ef825df64c799"} Jul 30 21:05:38.395288 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:38.395175 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff6cbd2793280acdd65b19f270f335fbfb124850077f96cf0d1ef825df64c799" Jul 30 21:05:50.330087 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.322286 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx"] Jul 30 21:05:50.330087 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.322640 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" podUID="09f483bf-4828-4a49-ab1a-ef46e7024a73" containerName="payload-processing" containerID="cri-o://75e4bf769a862f767ed85e6b843fca02c9a9aad8277f85c6b3b47ce4b0e5c46b" gracePeriod=30 Jul 30 21:05:50.338564 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.338144 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b"] Jul 30 21:05:50.338564 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.338407 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" podUID="bc8bddab-d42e-4a66-8f55-7dea2eea56fc" containerName="payload-pre-processing" containerID="cri-o://9ebc5badebe3771ab363d1a947ea200ba0285054f1e3f4d1d4b2d390f5abcc43" gracePeriod=30 Jul 30 21:05:50.464206 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.464170 2575 generic.go:356] "Generic (PLEG): container finished" podID="09f483bf-4828-4a49-ab1a-ef46e7024a73" containerID="75e4bf769a862f767ed85e6b843fca02c9a9aad8277f85c6b3b47ce4b0e5c46b" exitCode=0 Jul 30 21:05:50.464373 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.464241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" event={"ID":"09f483bf-4828-4a49-ab1a-ef46e7024a73","Type":"ContainerDied","Data":"75e4bf769a862f767ed85e6b843fca02c9a9aad8277f85c6b3b47ce4b0e5c46b"} Jul 30 21:05:50.466162 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.466122 2575 generic.go:356] "Generic (PLEG): container finished" podID="bc8bddab-d42e-4a66-8f55-7dea2eea56fc" containerID="9ebc5badebe3771ab363d1a947ea200ba0285054f1e3f4d1d4b2d390f5abcc43" exitCode=0 Jul 30 21:05:50.466301 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.466206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" event={"ID":"bc8bddab-d42e-4a66-8f55-7dea2eea56fc","Type":"ContainerDied","Data":"9ebc5badebe3771ab363d1a947ea200ba0285054f1e3f4d1d4b2d390f5abcc43"} Jul 30 21:05:50.497777 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.497495 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7"] Jul 30 21:05:50.499987 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.499964 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6e507a-5kkx7"] Jul 30 21:05:50.588946 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.588882 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:05:50.625247 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.625219 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:05:50.742317 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.742265 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-plugins-config-volume\") pod \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\" (UID: \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\") " Jul 30 21:05:50.742317 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.742323 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/09f483bf-4828-4a49-ab1a-ef46e7024a73-kube-api-access-cnnjv\") pod \"09f483bf-4828-4a49-ab1a-ef46e7024a73\" (UID: \"09f483bf-4828-4a49-ab1a-ef46e7024a73\") " Jul 30 21:05:50.742555 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.742436 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdkft\" (UniqueName: \"kubernetes.io/projected/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-kube-api-access-mdkft\") pod \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\" (UID: \"bc8bddab-d42e-4a66-8f55-7dea2eea56fc\") " Jul 30 21:05:50.742555 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.742469 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f483bf-4828-4a49-ab1a-ef46e7024a73-plugins-config-volume\") pod \"09f483bf-4828-4a49-ab1a-ef46e7024a73\" (UID: \"09f483bf-4828-4a49-ab1a-ef46e7024a73\") " Jul 30 21:05:50.742675 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.742650 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc8bddab-d42e-4a66-8f55-7dea2eea56fc" (UID: "bc8bddab-d42e-4a66-8f55-7dea2eea56fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:05:50.742918 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.742884 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:50.743082 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.742935 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f483bf-4828-4a49-ab1a-ef46e7024a73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09f483bf-4828-4a49-ab1a-ef46e7024a73" (UID: "09f483bf-4828-4a49-ab1a-ef46e7024a73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:05:50.744497 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.744477 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-kube-api-access-mdkft" (OuterVolumeSpecName: "kube-api-access-mdkft") pod "bc8bddab-d42e-4a66-8f55-7dea2eea56fc" (UID: "bc8bddab-d42e-4a66-8f55-7dea2eea56fc"). InnerVolumeSpecName "kube-api-access-mdkft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:05:50.744588 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.744511 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f483bf-4828-4a49-ab1a-ef46e7024a73-kube-api-access-cnnjv" (OuterVolumeSpecName: "kube-api-access-cnnjv") pod "09f483bf-4828-4a49-ab1a-ef46e7024a73" (UID: "09f483bf-4828-4a49-ab1a-ef46e7024a73"). InnerVolumeSpecName "kube-api-access-cnnjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:05:50.844434 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.844342 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdkft\" (UniqueName: \"kubernetes.io/projected/bc8bddab-d42e-4a66-8f55-7dea2eea56fc-kube-api-access-mdkft\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:50.844434 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.844377 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f483bf-4828-4a49-ab1a-ef46e7024a73-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:50.844434 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:50.844389 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/09f483bf-4828-4a49-ab1a-ef46e7024a73-kube-api-access-cnnjv\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:05:51.473663 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.473629 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" Jul 30 21:05:51.474160 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.473660 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx" event={"ID":"09f483bf-4828-4a49-ab1a-ef46e7024a73","Type":"ContainerDied","Data":"fc16c4c03b751233ccc4637b53f118b29868524924576370cab2cb6a861caebb"} Jul 30 21:05:51.474160 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.473740 2575 scope.go:117] "RemoveContainer" containerID="75e4bf769a862f767ed85e6b843fca02c9a9aad8277f85c6b3b47ce4b0e5c46b" Jul 30 21:05:51.475179 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.475153 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" Jul 30 21:05:51.475262 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.475206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b" event={"ID":"bc8bddab-d42e-4a66-8f55-7dea2eea56fc","Type":"ContainerDied","Data":"a2e3c9b835da074737246e5ccc8e58d9f860a8b79e427958df9fd893b4fdbd71"} Jul 30 21:05:51.484677 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.484654 2575 scope.go:117] "RemoveContainer" containerID="9ebc5badebe3771ab363d1a947ea200ba0285054f1e3f4d1d4b2d390f5abcc43" Jul 30 21:05:51.501914 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.501879 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b"] Jul 30 21:05:51.505550 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.505517 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6e507a-6f8f5bf596-wzv2b"] Jul 30 21:05:51.519629 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.519593 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx"] Jul 30 21:05:51.525713 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:51.525678 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6e507a-6945548d74-mbxzx"] Jul 30 21:05:52.013929 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:52.013895 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09f483bf-4828-4a49-ab1a-ef46e7024a73" path="/var/lib/kubelet/pods/09f483bf-4828-4a49-ab1a-ef46e7024a73/volumes" Jul 30 21:05:52.014286 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:52.014270 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b84222f-097c-479b-9704-a62df77a6bbd" path="/var/lib/kubelet/pods/0b84222f-097c-479b-9704-a62df77a6bbd/volumes" Jul 30 21:05:52.014608 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:05:52.014595 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc8bddab-d42e-4a66-8f55-7dea2eea56fc" path="/var/lib/kubelet/pods/bc8bddab-d42e-4a66-8f55-7dea2eea56fc/volumes" Jul 30 21:06:02.259489 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.259403 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:06:02.271205 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.271174 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:06:02.446098 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446045 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l"] Jul 30 21:06:02.446599 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446574 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b84222f-097c-479b-9704-a62df77a6bbd" containerName="revoke-keys" Jul 30 21:06:02.446800 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446766 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b84222f-097c-479b-9704-a62df77a6bbd" containerName="revoke-keys" Jul 30 21:06:02.446800 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446796 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8bddab-d42e-4a66-8f55-7dea2eea56fc" containerName="payload-pre-processing" Jul 30 21:06:02.446957 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446806 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8bddab-d42e-4a66-8f55-7dea2eea56fc" containerName="payload-pre-processing" Jul 30 21:06:02.446957 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446840 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f483bf-4828-4a49-ab1a-ef46e7024a73" containerName="payload-processing" Jul 30 21:06:02.446957 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446849 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f483bf-4828-4a49-ab1a-ef46e7024a73" containerName="payload-processing" Jul 30 21:06:02.447159 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446968 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="09f483bf-4828-4a49-ab1a-ef46e7024a73" containerName="payload-processing" Jul 30 21:06:02.447159 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446983 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8bddab-d42e-4a66-8f55-7dea2eea56fc" containerName="payload-pre-processing" Jul 30 21:06:02.447159 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.446995 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b84222f-097c-479b-9704-a62df77a6bbd" containerName="revoke-keys" Jul 30 21:06:02.451783 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.451759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:02.454566 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.454546 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-55df2b\"" Jul 30 21:06:02.466099 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.465959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l"] Jul 30 21:06:02.467450 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.467425 2575 scope.go:117] "RemoveContainer" containerID="e7db63af1fc59cdfc1a4dada7559c2aad6a85c506a338cff136d716a92df8927" Jul 30 21:06:02.482837 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.482807 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f"] Jul 30 21:06:02.491358 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.491323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:02.497756 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.497720 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-55df2b-dockercfg-86wl9\"" Jul 30 21:06:02.503042 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.502273 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f"] Jul 30 21:06:02.552737 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.552702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdf8\" (UniqueName: \"kubernetes.io/projected/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-kube-api-access-srdf8\") pod \"payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l\" (UID: \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:02.552958 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.552742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l\" (UID: \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:02.653809 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.653768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08435e1f-ac77-4181-a9b0-18a8d4b9f984-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f\" (UID: \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:02.654011 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.653842 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdf8\" (UniqueName: \"kubernetes.io/projected/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-kube-api-access-srdf8\") pod \"payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l\" (UID: \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:02.654011 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.653867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l\" (UID: \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:02.654011 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.653907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979lm\" (UniqueName: \"kubernetes.io/projected/08435e1f-ac77-4181-a9b0-18a8d4b9f984-kube-api-access-979lm\") pod \"payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f\" (UID: \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:02.654628 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.654604 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l\" (UID: \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:02.662888 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.662855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdf8\" (UniqueName: \"kubernetes.io/projected/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-kube-api-access-srdf8\") pod \"payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l\" (UID: \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:02.755026 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.754986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08435e1f-ac77-4181-a9b0-18a8d4b9f984-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f\" (UID: \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:02.755249 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.755111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979lm\" (UniqueName: \"kubernetes.io/projected/08435e1f-ac77-4181-a9b0-18a8d4b9f984-kube-api-access-979lm\") pod \"payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f\" (UID: \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:02.755727 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.755703 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08435e1f-ac77-4181-a9b0-18a8d4b9f984-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f\" (UID: \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:02.762521 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.762437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:02.763559 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.763539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-979lm\" (UniqueName: \"kubernetes.io/projected/08435e1f-ac77-4181-a9b0-18a8d4b9f984-kube-api-access-979lm\") pod \"payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f\" (UID: \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:02.810219 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.810187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:02.921653 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.921620 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l"] Jul 30 21:06:02.978384 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:02.978355 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f"] Jul 30 21:06:02.980756 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:06:02.980728 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08435e1f_ac77_4181_a9b0_18a8d4b9f984.slice/crio-ffb40657583a430f95c13717e6c79afc97cf08a782786a24dfa200f490dec55f WatchSource:0}: Error finding container ffb40657583a430f95c13717e6c79afc97cf08a782786a24dfa200f490dec55f: Status 404 returned error can't find the container with id ffb40657583a430f95c13717e6c79afc97cf08a782786a24dfa200f490dec55f Jul 30 21:06:03.547103 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:03.547042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" event={"ID":"a2e3862b-2010-49f5-af97-d9b4cc8c88b5","Type":"ContainerStarted","Data":"46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447"} Jul 30 21:06:03.547766 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:03.547127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" event={"ID":"a2e3862b-2010-49f5-af97-d9b4cc8c88b5","Type":"ContainerStarted","Data":"626af5180e13f1af8db0e2e9795c6c27fe5bd9951f19190f3ff751754e5e951b"} Jul 30 21:06:03.547766 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:03.547254 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:03.548835 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:03.548810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" event={"ID":"08435e1f-ac77-4181-a9b0-18a8d4b9f984","Type":"ContainerStarted","Data":"43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e"} Jul 30 21:06:03.548970 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:03.548858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" event={"ID":"08435e1f-ac77-4181-a9b0-18a8d4b9f984","Type":"ContainerStarted","Data":"ffb40657583a430f95c13717e6c79afc97cf08a782786a24dfa200f490dec55f"} Jul 30 21:06:03.548970 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:03.548918 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:03.563759 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:03.563705 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" podStartSLOduration=1.563687851 podStartE2EDuration="1.563687851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:06:03.562297058 +0000 UTC m=+2702.027435417" watchObservedRunningTime="2026-07-30 21:06:03.563687851 +0000 UTC m=+2702.028826198" Jul 30 21:06:03.581150 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:03.581093 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" podStartSLOduration=1.5810763159999999 podStartE2EDuration="1.581076316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:06:03.579001781 +0000 UTC m=+2702.044140131" watchObservedRunningTime="2026-07-30 21:06:03.581076316 +0000 UTC m=+2702.046214655" Jul 30 21:06:14.556559 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:14.556528 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:06:14.556974 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:14.556693 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:06:31.534954 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.534916 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm"] Jul 30 21:06:31.538856 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.538827 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:31.541777 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.541751 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-e226df-dockercfg-ddnmj\"" Jul 30 21:06:31.541933 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.541791 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-e226df\"" Jul 30 21:06:31.551944 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.551919 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp"] Jul 30 21:06:31.555411 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.555390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm"] Jul 30 21:06:31.555542 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.555508 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:31.574129 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.574102 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp"] Jul 30 21:06:31.622826 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.622787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfpb\" (UniqueName: \"kubernetes.io/projected/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-kube-api-access-hkfpb\") pod \"payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm\" (UID: \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:31.622826 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.622831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qlf\" (UniqueName: \"kubernetes.io/projected/de003a9f-19c2-42c3-93ba-a97b658185cb-kube-api-access-d5qlf\") pod \"payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp\" (UID: \"de003a9f-19c2-42c3-93ba-a97b658185cb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:31.623082 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.622940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm\" (UID: \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:31.623082 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.622970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de003a9f-19c2-42c3-93ba-a97b658185cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp\" (UID: \"de003a9f-19c2-42c3-93ba-a97b658185cb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:31.723619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.723578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfpb\" (UniqueName: \"kubernetes.io/projected/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-kube-api-access-hkfpb\") pod \"payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm\" (UID: \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:31.723619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.723618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qlf\" (UniqueName: \"kubernetes.io/projected/de003a9f-19c2-42c3-93ba-a97b658185cb-kube-api-access-d5qlf\") pod \"payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp\" (UID: \"de003a9f-19c2-42c3-93ba-a97b658185cb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:31.723851 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.723666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm\" (UID: \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:31.723851 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.723685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de003a9f-19c2-42c3-93ba-a97b658185cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp\" (UID: \"de003a9f-19c2-42c3-93ba-a97b658185cb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:31.724688 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.724659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm\" (UID: \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:31.724778 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.724696 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de003a9f-19c2-42c3-93ba-a97b658185cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp\" (UID: \"de003a9f-19c2-42c3-93ba-a97b658185cb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:31.731801 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.731776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qlf\" (UniqueName: \"kubernetes.io/projected/de003a9f-19c2-42c3-93ba-a97b658185cb-kube-api-access-d5qlf\") pod \"payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp\" (UID: \"de003a9f-19c2-42c3-93ba-a97b658185cb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:31.731960 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.731937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfpb\" (UniqueName: \"kubernetes.io/projected/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-kube-api-access-hkfpb\") pod \"payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm\" (UID: \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:31.851708 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.851617 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:31.865095 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:31.865066 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:32.019588 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.019342 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm"] Jul 30 21:06:32.247742 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.247713 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp"] Jul 30 21:06:32.250581 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:06:32.250537 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde003a9f_19c2_42c3_93ba_a97b658185cb.slice/crio-014b27365fd9204fceecd86b076f3d1cabc25555cb220eb87d9e6b0950d85a1c WatchSource:0}: Error finding container 014b27365fd9204fceecd86b076f3d1cabc25555cb220eb87d9e6b0950d85a1c: Status 404 returned error can't find the container with id 014b27365fd9204fceecd86b076f3d1cabc25555cb220eb87d9e6b0950d85a1c Jul 30 21:06:32.709689 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.709657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" event={"ID":"de003a9f-19c2-42c3-93ba-a97b658185cb","Type":"ContainerStarted","Data":"e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344"} Jul 30 21:06:32.710176 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.709716 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" event={"ID":"de003a9f-19c2-42c3-93ba-a97b658185cb","Type":"ContainerStarted","Data":"014b27365fd9204fceecd86b076f3d1cabc25555cb220eb87d9e6b0950d85a1c"} Jul 30 21:06:32.710176 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.709833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:32.711213 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.711189 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" event={"ID":"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae","Type":"ContainerStarted","Data":"ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f"} Jul 30 21:06:32.711339 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.711230 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" event={"ID":"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae","Type":"ContainerStarted","Data":"3ad5e5b836d38d20f9a31aaff41af4974df52883b922dff0d3b14086ae0db873"} Jul 30 21:06:32.711339 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.711311 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:06:32.729046 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.728993 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" podStartSLOduration=1.7289768699999999 podStartE2EDuration="1.72897687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:06:32.726468998 +0000 UTC m=+2731.191607350" watchObservedRunningTime="2026-07-30 21:06:32.72897687 +0000 UTC m=+2731.194115196" Jul 30 21:06:32.743188 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:32.743140 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" podStartSLOduration=1.7431228970000001 podStartE2EDuration="1.743122897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:06:32.742586993 +0000 UTC m=+2731.207725341" watchObservedRunningTime="2026-07-30 21:06:32.743122897 +0000 UTC m=+2731.208261244" Jul 30 21:06:43.718150 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:43.718119 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:06:43.718588 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:06:43.718188 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:10:02.552016 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:10:02.551981 2575 scope.go:117] "RemoveContainer" containerID="403def8a49492bd73b0042c4d719b6c8bcfdedd40b7ed0ea707115216e72655c" Jul 30 21:10:02.561280 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:10:02.561260 2575 scope.go:117] "RemoveContainer" containerID="da995bf7b8dfdc980178335ba6bb95b3b34ff397e1efdf99ce3033ceb1595390" Jul 30 21:11:02.295390 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:11:02.295315 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:11:02.310259 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:11:02.310232 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:12:02.603224 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:12:02.603131 2575 scope.go:117] "RemoveContainer" containerID="d05e70482afee2dd050e12a5408d7fa083e3f79cdbe0857a0c39746eb3bdf520" Jul 30 21:12:02.612124 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:12:02.612103 2575 scope.go:117] "RemoveContainer" containerID="d95763c3baf7c0391ac4b1ff28f4a1f86109df77b680abfcfb88ff7ddfcd8942" Jul 30 21:15:00.152116 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.151969 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz"] Jul 30 21:15:00.155617 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.155594 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5"] Jul 30 21:15:00.155750 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.155714 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz" Jul 30 21:15:00.158297 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.158268 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:15:00.159165 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.159147 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf"] Jul 30 21:15:00.159260 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.159245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5" Jul 30 21:15:00.162390 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.162365 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf" Jul 30 21:15:00.167582 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.167556 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz"] Jul 30 21:15:00.168304 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.168286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz" Jul 30 21:15:00.173574 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.173544 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf"] Jul 30 21:15:00.184783 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.184744 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5" Jul 30 21:15:00.190550 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.190521 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5"] Jul 30 21:15:00.223592 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.223561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf" Jul 30 21:15:00.364901 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.364860 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz"] Jul 30 21:15:00.366504 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:15:00.366477 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6219b979_4c25_4f59_aa34_7e521727bc63.slice/crio-802724151cc3024e3461a69f47c2956d13df2d0fe853488c66ca2fb9166154ba WatchSource:0}: Error finding container 802724151cc3024e3461a69f47c2956d13df2d0fe853488c66ca2fb9166154ba: Status 404 returned error can't find the container with id 802724151cc3024e3461a69f47c2956d13df2d0fe853488c66ca2fb9166154ba Jul 30 21:15:00.411376 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.411350 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf"] Jul 30 21:15:00.412200 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:15:00.412177 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1686c1e9_7d74_4bef_96ba_8bdb2043baf7.slice/crio-3f3d9fba411c8b03533cf88586473c392a5367b64f2a5daac50be9380cb14546 WatchSource:0}: Error finding container 3f3d9fba411c8b03533cf88586473c392a5367b64f2a5daac50be9380cb14546: Status 404 returned error can't find the container with id 3f3d9fba411c8b03533cf88586473c392a5367b64f2a5daac50be9380cb14546 Jul 30 21:15:00.542389 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.542356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz" event={"ID":"6219b979-4c25-4f59-aa34-7e521727bc63","Type":"ContainerStarted","Data":"802724151cc3024e3461a69f47c2956d13df2d0fe853488c66ca2fb9166154ba"} Jul 30 21:15:00.543585 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.543556 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf" event={"ID":"1686c1e9-7d74-4bef-96ba-8bdb2043baf7","Type":"ContainerStarted","Data":"3f3d9fba411c8b03533cf88586473c392a5367b64f2a5daac50be9380cb14546"} Jul 30 21:15:00.591790 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:00.591761 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5"] Jul 30 21:15:00.592986 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:15:00.592965 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49514f54_b1eb_42d2_95dd_897da46ff7af.slice/crio-457ce9d8453ee85bd1214e6f779722ade198447755695d42a3a9d12bcccf87f3 WatchSource:0}: Error finding container 457ce9d8453ee85bd1214e6f779722ade198447755695d42a3a9d12bcccf87f3: Status 404 returned error can't find the container with id 457ce9d8453ee85bd1214e6f779722ade198447755695d42a3a9d12bcccf87f3 Jul 30 21:15:01.550070 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:01.550024 2575 generic.go:356] "Generic (PLEG): container finished" podID="1686c1e9-7d74-4bef-96ba-8bdb2043baf7" containerID="515dad15e81565ce5f1982583a64b8932ac70ca99303f60c646ac32ac09a53d8" exitCode=0 Jul 30 21:15:01.550502 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:01.550097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf" event={"ID":"1686c1e9-7d74-4bef-96ba-8bdb2043baf7","Type":"ContainerDied","Data":"515dad15e81565ce5f1982583a64b8932ac70ca99303f60c646ac32ac09a53d8"} Jul 30 21:15:01.551684 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:01.551651 2575 generic.go:356] "Generic (PLEG): container finished" podID="49514f54-b1eb-42d2-95dd-897da46ff7af" containerID="93cd140bc25bfbbb1540f44c36f6bc1d8594436b76101fffe8a76db38d390c44" exitCode=0 Jul 30 21:15:01.551802 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:01.551733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5" event={"ID":"49514f54-b1eb-42d2-95dd-897da46ff7af","Type":"ContainerDied","Data":"93cd140bc25bfbbb1540f44c36f6bc1d8594436b76101fffe8a76db38d390c44"} Jul 30 21:15:01.551802 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:01.551787 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5" event={"ID":"49514f54-b1eb-42d2-95dd-897da46ff7af","Type":"ContainerStarted","Data":"457ce9d8453ee85bd1214e6f779722ade198447755695d42a3a9d12bcccf87f3"} Jul 30 21:15:01.553125 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:01.553106 2575 generic.go:356] "Generic (PLEG): container finished" podID="6219b979-4c25-4f59-aa34-7e521727bc63" containerID="0a1be0bd8b2a04aaf311217c213c1191704727c98bad3b18377133062f57bc40" exitCode=0 Jul 30 21:15:01.553222 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:01.553176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz" event={"ID":"6219b979-4c25-4f59-aa34-7e521727bc63","Type":"ContainerDied","Data":"0a1be0bd8b2a04aaf311217c213c1191704727c98bad3b18377133062f57bc40"} Jul 30 21:15:02.719046 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:02.719018 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf" Jul 30 21:15:02.743209 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:02.743179 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5" Jul 30 21:15:02.747533 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:02.747503 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz" Jul 30 21:15:03.565514 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.565483 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5" Jul 30 21:15:03.565739 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.565523 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5" event={"ID":"49514f54-b1eb-42d2-95dd-897da46ff7af","Type":"ContainerDied","Data":"457ce9d8453ee85bd1214e6f779722ade198447755695d42a3a9d12bcccf87f3"} Jul 30 21:15:03.565739 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.565569 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="457ce9d8453ee85bd1214e6f779722ade198447755695d42a3a9d12bcccf87f3" Jul 30 21:15:03.566908 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.566891 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz" Jul 30 21:15:03.566991 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.566927 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-6t8xz" event={"ID":"6219b979-4c25-4f59-aa34-7e521727bc63","Type":"ContainerDied","Data":"802724151cc3024e3461a69f47c2956d13df2d0fe853488c66ca2fb9166154ba"} Jul 30 21:15:03.566991 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.566971 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="802724151cc3024e3461a69f47c2956d13df2d0fe853488c66ca2fb9166154ba" Jul 30 21:15:03.568298 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.568278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf" event={"ID":"1686c1e9-7d74-4bef-96ba-8bdb2043baf7","Type":"ContainerDied","Data":"3f3d9fba411c8b03533cf88586473c392a5367b64f2a5daac50be9380cb14546"} Jul 30 21:15:03.568425 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.568299 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf" Jul 30 21:15:03.568425 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.568312 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f3d9fba411c8b03533cf88586473c392a5367b64f2a5daac50be9380cb14546" Jul 30 21:15:03.823598 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.823505 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94"] Jul 30 21:15:03.826515 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:03.826490 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757390-pwc94"] Jul 30 21:15:04.013944 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:04.013910 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5619676f-bc7a-4291-8050-4593f589263a" path="/var/lib/kubelet/pods/5619676f-bc7a-4291-8050-4593f589263a/volumes" Jul 30 21:15:52.796333 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796290 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b"] Jul 30 21:15:52.796798 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796710 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1686c1e9-7d74-4bef-96ba-8bdb2043baf7" containerName="cleanup" Jul 30 21:15:52.796798 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796722 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1686c1e9-7d74-4bef-96ba-8bdb2043baf7" containerName="cleanup" Jul 30 21:15:52.796798 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796730 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6219b979-4c25-4f59-aa34-7e521727bc63" containerName="cleanup" Jul 30 21:15:52.796798 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796737 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6219b979-4c25-4f59-aa34-7e521727bc63" containerName="cleanup" Jul 30 21:15:52.796798 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796751 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49514f54-b1eb-42d2-95dd-897da46ff7af" containerName="cleanup" Jul 30 21:15:52.796798 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796757 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49514f54-b1eb-42d2-95dd-897da46ff7af" containerName="cleanup" Jul 30 21:15:52.796990 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796829 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49514f54-b1eb-42d2-95dd-897da46ff7af" containerName="cleanup" Jul 30 21:15:52.796990 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796837 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6219b979-4c25-4f59-aa34-7e521727bc63" containerName="cleanup" Jul 30 21:15:52.796990 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.796846 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1686c1e9-7d74-4bef-96ba-8bdb2043baf7" containerName="cleanup" Jul 30 21:15:52.800018 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.799998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:52.802935 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.802914 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-720e63-dockercfg-7flb2\"" Jul 30 21:15:52.803543 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.803526 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-720e63\"" Jul 30 21:15:52.816044 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.816008 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b"] Jul 30 21:15:52.897239 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.897201 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh"] Jul 30 21:15:52.897614 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.897588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa363f9c-f9aa-4379-85a9-d5466bea76fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b\" (UID: \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:52.897725 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.897656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzjs\" (UniqueName: \"kubernetes.io/projected/fa363f9c-f9aa-4379-85a9-d5466bea76fd-kube-api-access-cvzjs\") pod \"payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b\" (UID: \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:52.901407 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.901387 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:52.924462 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.924421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh"] Jul 30 21:15:52.998778 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.998743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k8lf\" (UniqueName: \"kubernetes.io/projected/207c9c60-3520-4da2-8dc9-cc9ea1b29929-kube-api-access-4k8lf\") pod \"payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh\" (UID: \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:52.998971 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.998785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa363f9c-f9aa-4379-85a9-d5466bea76fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b\" (UID: \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:52.998971 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.998867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c9c60-3520-4da2-8dc9-cc9ea1b29929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh\" (UID: \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:52.998971 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.998912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzjs\" (UniqueName: \"kubernetes.io/projected/fa363f9c-f9aa-4379-85a9-d5466bea76fd-kube-api-access-cvzjs\") pod \"payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b\" (UID: \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:52.999604 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:52.999582 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa363f9c-f9aa-4379-85a9-d5466bea76fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b\" (UID: \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:53.009674 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.009643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzjs\" (UniqueName: \"kubernetes.io/projected/fa363f9c-f9aa-4379-85a9-d5466bea76fd-kube-api-access-cvzjs\") pod \"payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b\" (UID: \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:53.099937 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.099808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k8lf\" (UniqueName: \"kubernetes.io/projected/207c9c60-3520-4da2-8dc9-cc9ea1b29929-kube-api-access-4k8lf\") pod \"payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh\" (UID: \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:53.100563 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.100533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c9c60-3520-4da2-8dc9-cc9ea1b29929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh\" (UID: \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:53.101280 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.101259 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c9c60-3520-4da2-8dc9-cc9ea1b29929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh\" (UID: \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:53.109445 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.109419 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k8lf\" (UniqueName: \"kubernetes.io/projected/207c9c60-3520-4da2-8dc9-cc9ea1b29929-kube-api-access-4k8lf\") pod \"payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh\" (UID: \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:53.111196 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.111179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:53.211361 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.211215 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:53.260942 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.260326 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b"] Jul 30 21:15:53.263014 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:15:53.262983 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa363f9c_f9aa_4379_85a9_d5466bea76fd.slice/crio-ef0a3e75531cbb5c082ef06bd82151ce47fdcdd8a496f0f31a84f6659fb6d78a WatchSource:0}: Error finding container ef0a3e75531cbb5c082ef06bd82151ce47fdcdd8a496f0f31a84f6659fb6d78a: Status 404 returned error can't find the container with id ef0a3e75531cbb5c082ef06bd82151ce47fdcdd8a496f0f31a84f6659fb6d78a Jul 30 21:15:53.376605 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.376573 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh"] Jul 30 21:15:53.378157 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:15:53.378127 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207c9c60_3520_4da2_8dc9_cc9ea1b29929.slice/crio-2e32d37bbec86ac9fff8e1da53db277b393715532d90d31b5feb44f3d8c1bea3 WatchSource:0}: Error finding container 2e32d37bbec86ac9fff8e1da53db277b393715532d90d31b5feb44f3d8c1bea3: Status 404 returned error can't find the container with id 2e32d37bbec86ac9fff8e1da53db277b393715532d90d31b5feb44f3d8c1bea3 Jul 30 21:15:53.868282 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.868246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" event={"ID":"fa363f9c-f9aa-4379-85a9-d5466bea76fd","Type":"ContainerStarted","Data":"82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61"} Jul 30 21:15:53.868785 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.868317 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" event={"ID":"fa363f9c-f9aa-4379-85a9-d5466bea76fd","Type":"ContainerStarted","Data":"ef0a3e75531cbb5c082ef06bd82151ce47fdcdd8a496f0f31a84f6659fb6d78a"} Jul 30 21:15:53.868785 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.868429 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:15:53.870028 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.870003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" event={"ID":"207c9c60-3520-4da2-8dc9-cc9ea1b29929","Type":"ContainerStarted","Data":"962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98"} Jul 30 21:15:53.870169 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.870045 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" event={"ID":"207c9c60-3520-4da2-8dc9-cc9ea1b29929","Type":"ContainerStarted","Data":"2e32d37bbec86ac9fff8e1da53db277b393715532d90d31b5feb44f3d8c1bea3"} Jul 30 21:15:53.870169 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.870086 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:15:53.886303 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.886253 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" podStartSLOduration=1.886236156 podStartE2EDuration="1.886236156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:15:53.883754585 +0000 UTC m=+3292.348892935" watchObservedRunningTime="2026-07-30 21:15:53.886236156 +0000 UTC m=+3292.351374504" Jul 30 21:15:53.900488 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:15:53.900427 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" podStartSLOduration=1.900410229 podStartE2EDuration="1.900410229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:15:53.899241794 +0000 UTC m=+3292.364380167" watchObservedRunningTime="2026-07-30 21:15:53.900410229 +0000 UTC m=+3292.365548581" Jul 30 21:16:02.332548 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:02.332518 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:16:02.348713 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:02.348687 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:16:02.671868 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:02.667114 2575 scope.go:117] "RemoveContainer" containerID="5409286269ff6a269e9f020cc1a8fc8137f434f8031703a269ca6af32b6b518a" Jul 30 21:16:04.877804 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:04.877770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:16:04.878217 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:04.877938 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:16:21.758772 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.758727 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4"] Jul 30 21:16:21.762899 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.762875 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:21.765682 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.765655 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6967f3\"" Jul 30 21:16:21.765838 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.765752 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6967f3-dockercfg-6btr2\"" Jul 30 21:16:21.774618 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.774586 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4"] Jul 30 21:16:21.790342 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.790300 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x"] Jul 30 21:16:21.794382 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.794354 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:21.805582 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.805554 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x"] Jul 30 21:16:21.863727 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.863685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872d4680-b538-48c7-bb03-d63acafcb058-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4\" (UID: \"872d4680-b538-48c7-bb03-d63acafcb058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:21.863957 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.863808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7489t\" (UniqueName: \"kubernetes.io/projected/9530be12-a2d2-41f8-a20c-1801ec6aab61-kube-api-access-7489t\") pod \"payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x\" (UID: \"9530be12-a2d2-41f8-a20c-1801ec6aab61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:21.863957 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.863872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbq2t\" (UniqueName: \"kubernetes.io/projected/872d4680-b538-48c7-bb03-d63acafcb058-kube-api-access-gbq2t\") pod \"payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4\" (UID: \"872d4680-b538-48c7-bb03-d63acafcb058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:21.864151 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.864119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9530be12-a2d2-41f8-a20c-1801ec6aab61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x\" (UID: \"9530be12-a2d2-41f8-a20c-1801ec6aab61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:21.965673 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.965575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9530be12-a2d2-41f8-a20c-1801ec6aab61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x\" (UID: \"9530be12-a2d2-41f8-a20c-1801ec6aab61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:21.965858 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.965730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872d4680-b538-48c7-bb03-d63acafcb058-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4\" (UID: \"872d4680-b538-48c7-bb03-d63acafcb058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:21.965858 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.965797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7489t\" (UniqueName: \"kubernetes.io/projected/9530be12-a2d2-41f8-a20c-1801ec6aab61-kube-api-access-7489t\") pod \"payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x\" (UID: \"9530be12-a2d2-41f8-a20c-1801ec6aab61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:21.965858 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.965831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbq2t\" (UniqueName: \"kubernetes.io/projected/872d4680-b538-48c7-bb03-d63acafcb058-kube-api-access-gbq2t\") pod \"payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4\" (UID: \"872d4680-b538-48c7-bb03-d63acafcb058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:21.966411 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.966392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9530be12-a2d2-41f8-a20c-1801ec6aab61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x\" (UID: \"9530be12-a2d2-41f8-a20c-1801ec6aab61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:21.966603 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.966583 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872d4680-b538-48c7-bb03-d63acafcb058-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4\" (UID: \"872d4680-b538-48c7-bb03-d63acafcb058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:21.975901 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.975869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbq2t\" (UniqueName: \"kubernetes.io/projected/872d4680-b538-48c7-bb03-d63acafcb058-kube-api-access-gbq2t\") pod \"payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4\" (UID: \"872d4680-b538-48c7-bb03-d63acafcb058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:21.976146 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:21.976126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7489t\" (UniqueName: \"kubernetes.io/projected/9530be12-a2d2-41f8-a20c-1801ec6aab61-kube-api-access-7489t\") pod \"payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x\" (UID: \"9530be12-a2d2-41f8-a20c-1801ec6aab61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:22.074983 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:22.074889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:22.109442 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:22.109397 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:22.242890 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:22.242853 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4"] Jul 30 21:16:22.243309 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:16:22.243271 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872d4680_b538_48c7_bb03_d63acafcb058.slice/crio-6a4e6e308242aac502ab9d4ea5deb01973c15498381778e39d1f0e69ed24f9da WatchSource:0}: Error finding container 6a4e6e308242aac502ab9d4ea5deb01973c15498381778e39d1f0e69ed24f9da: Status 404 returned error can't find the container with id 6a4e6e308242aac502ab9d4ea5deb01973c15498381778e39d1f0e69ed24f9da Jul 30 21:16:22.276832 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:22.276801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x"] Jul 30 21:16:22.283393 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:16:22.283360 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9530be12_a2d2_41f8_a20c_1801ec6aab61.slice/crio-0e40432d11c6393162015827a9e0ed55fa241c39b324803dbeb9497a20d9b3cf WatchSource:0}: Error finding container 0e40432d11c6393162015827a9e0ed55fa241c39b324803dbeb9497a20d9b3cf: Status 404 returned error can't find the container with id 0e40432d11c6393162015827a9e0ed55fa241c39b324803dbeb9497a20d9b3cf Jul 30 21:16:23.046314 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:23.046259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" event={"ID":"872d4680-b538-48c7-bb03-d63acafcb058","Type":"ContainerStarted","Data":"17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405"} Jul 30 21:16:23.046757 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:23.046342 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:23.046757 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:23.046367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" event={"ID":"872d4680-b538-48c7-bb03-d63acafcb058","Type":"ContainerStarted","Data":"6a4e6e308242aac502ab9d4ea5deb01973c15498381778e39d1f0e69ed24f9da"} Jul 30 21:16:23.048427 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:23.048395 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" event={"ID":"9530be12-a2d2-41f8-a20c-1801ec6aab61","Type":"ContainerStarted","Data":"f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01"} Jul 30 21:16:23.048592 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:23.048460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" event={"ID":"9530be12-a2d2-41f8-a20c-1801ec6aab61","Type":"ContainerStarted","Data":"0e40432d11c6393162015827a9e0ed55fa241c39b324803dbeb9497a20d9b3cf"} Jul 30 21:16:23.048592 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:23.048482 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:23.062045 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:23.061981 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" podStartSLOduration=2.061960525 podStartE2EDuration="2.061960525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:16:23.061571283 +0000 UTC m=+3321.526709644" watchObservedRunningTime="2026-07-30 21:16:23.061960525 +0000 UTC m=+3321.527098874" Jul 30 21:16:23.078933 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:23.078872 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" podStartSLOduration=2.078854845 podStartE2EDuration="2.078854845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:16:23.076454729 +0000 UTC m=+3321.541593078" watchObservedRunningTime="2026-07-30 21:16:23.078854845 +0000 UTC m=+3321.543993192" Jul 30 21:16:34.058131 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:34.058100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:16:34.058657 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:34.058171 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:16:51.973613 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:51.973578 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-69rsg_b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8/payload-processing/0.log" Jul 30 21:16:52.102592 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:16:52.102561 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x_9530be12-a2d2-41f8-a20c-1801ec6aab61/payload-processing/0.log" Jul 30 21:17:42.097908 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:42.097871 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh_207c9c60-3520-4da2-8dc9-cc9ea1b29929/payload-processing/0.log" Jul 30 21:17:42.219923 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:42.219886 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-69rsg_b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8/payload-processing/0.log" Jul 30 21:17:42.342028 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:42.342000 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x_9530be12-a2d2-41f8-a20c-1801ec6aab61/payload-processing/0.log" Jul 30 21:17:47.277538 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.277488 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8"] Jul 30 21:17:47.282898 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.282848 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:47.286600 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.286569 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx"] Jul 30 21:17:47.287726 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.287703 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e026c1\"" Jul 30 21:17:47.287726 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.287719 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e026c1-dockercfg-4qfm6\"" Jul 30 21:17:47.290992 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.290969 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:47.295444 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.295421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8"] Jul 30 21:17:47.304635 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.304608 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx"] Jul 30 21:17:47.332704 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.332669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8\" (UID: \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:47.332888 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.332709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgw4z\" (UniqueName: \"kubernetes.io/projected/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-kube-api-access-rgw4z\") pod \"payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8\" (UID: \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:47.332888 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.332779 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9593de9-4c07-4017-a234-4c07cd461a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx\" (UID: \"b9593de9-4c07-4017-a234-4c07cd461a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:47.332888 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.332801 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfmj\" (UniqueName: \"kubernetes.io/projected/b9593de9-4c07-4017-a234-4c07cd461a3f-kube-api-access-kwfmj\") pod \"payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx\" (UID: \"b9593de9-4c07-4017-a234-4c07cd461a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:47.434317 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.434276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgw4z\" (UniqueName: \"kubernetes.io/projected/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-kube-api-access-rgw4z\") pod \"payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8\" (UID: \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:47.434481 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.434382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9593de9-4c07-4017-a234-4c07cd461a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx\" (UID: \"b9593de9-4c07-4017-a234-4c07cd461a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:47.434481 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.434415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfmj\" (UniqueName: \"kubernetes.io/projected/b9593de9-4c07-4017-a234-4c07cd461a3f-kube-api-access-kwfmj\") pod \"payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx\" (UID: \"b9593de9-4c07-4017-a234-4c07cd461a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:47.434596 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.434548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8\" (UID: \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:47.435158 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.435135 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9593de9-4c07-4017-a234-4c07cd461a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx\" (UID: \"b9593de9-4c07-4017-a234-4c07cd461a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:47.435429 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.435407 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8\" (UID: \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:47.447902 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.447866 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgw4z\" (UniqueName: \"kubernetes.io/projected/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-kube-api-access-rgw4z\") pod \"payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8\" (UID: \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:47.447902 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.447897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfmj\" (UniqueName: \"kubernetes.io/projected/b9593de9-4c07-4017-a234-4c07cd461a3f-kube-api-access-kwfmj\") pod \"payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx\" (UID: \"b9593de9-4c07-4017-a234-4c07cd461a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:47.600734 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.600641 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:47.608915 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.608889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:47.780493 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.780460 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx"] Jul 30 21:17:47.781974 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:17:47.781947 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9593de9_4c07_4017_a234_4c07cd461a3f.slice/crio-fb8485e21ca05f9edfbdce675144584bb522881dc1a60c917b4f360bb408dc11 WatchSource:0}: Error finding container fb8485e21ca05f9edfbdce675144584bb522881dc1a60c917b4f360bb408dc11: Status 404 returned error can't find the container with id fb8485e21ca05f9edfbdce675144584bb522881dc1a60c917b4f360bb408dc11 Jul 30 21:17:47.966877 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:47.966841 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8"] Jul 30 21:17:47.968008 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:17:47.967975 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73d4ff0_3a00_4eb5_b8e2_163a7b7b5439.slice/crio-77d15873723e8ddf6cf08ac71639e44cb1706ec74c08fef29c4cc6949c19edfe WatchSource:0}: Error finding container 77d15873723e8ddf6cf08ac71639e44cb1706ec74c08fef29c4cc6949c19edfe: Status 404 returned error can't find the container with id 77d15873723e8ddf6cf08ac71639e44cb1706ec74c08fef29c4cc6949c19edfe Jul 30 21:17:48.541322 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:48.541286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" event={"ID":"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439","Type":"ContainerStarted","Data":"6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f"} Jul 30 21:17:48.541779 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:48.541346 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" event={"ID":"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439","Type":"ContainerStarted","Data":"77d15873723e8ddf6cf08ac71639e44cb1706ec74c08fef29c4cc6949c19edfe"} Jul 30 21:17:48.541779 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:48.541506 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:48.543134 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:48.543108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" event={"ID":"b9593de9-4c07-4017-a234-4c07cd461a3f","Type":"ContainerStarted","Data":"3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6"} Jul 30 21:17:48.543258 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:48.543153 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" event={"ID":"b9593de9-4c07-4017-a234-4c07cd461a3f","Type":"ContainerStarted","Data":"fb8485e21ca05f9edfbdce675144584bb522881dc1a60c917b4f360bb408dc11"} Jul 30 21:17:48.543258 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:48.543174 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:17:48.557619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:48.557561 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" podStartSLOduration=1.557545985 podStartE2EDuration="1.557545985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:17:48.556131736 +0000 UTC m=+3407.021270087" watchObservedRunningTime="2026-07-30 21:17:48.557545985 +0000 UTC m=+3407.022684333" Jul 30 21:17:48.573525 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:48.573475 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" podStartSLOduration=1.573458764 podStartE2EDuration="1.573458764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:17:48.5711433 +0000 UTC m=+3407.036281648" watchObservedRunningTime="2026-07-30 21:17:48.573458764 +0000 UTC m=+3407.038597112" Jul 30 21:17:59.551089 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:59.550976 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:17:59.551634 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:17:59.551161 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:18:11.162072 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.162020 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5"] Jul 30 21:18:11.166840 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.166822 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" Jul 30 21:18:11.169590 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.169562 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:18:11.173609 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.173578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5"] Jul 30 21:18:11.266894 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.266865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d41e11a-b58e-4063-a477-9ea2ae50fdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5\" (UID: \"5d41e11a-b58e-4063-a477-9ea2ae50fdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" Jul 30 21:18:11.367864 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.367817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d41e11a-b58e-4063-a477-9ea2ae50fdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5\" (UID: \"5d41e11a-b58e-4063-a477-9ea2ae50fdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" Jul 30 21:18:11.368502 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.368481 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d41e11a-b58e-4063-a477-9ea2ae50fdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5\" (UID: \"5d41e11a-b58e-4063-a477-9ea2ae50fdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" Jul 30 21:18:11.480672 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.480644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" Jul 30 21:18:11.629523 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.629487 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5"] Jul 30 21:18:11.630403 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:18:11.630368 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d41e11a_b58e_4063_a477_9ea2ae50fdd2.slice/crio-738fea561e5e43736f737aae47c7b1a67ea57d6470a98d82e503ca9888237ba7 WatchSource:0}: Error finding container 738fea561e5e43736f737aae47c7b1a67ea57d6470a98d82e503ca9888237ba7: Status 404 returned error can't find the container with id 738fea561e5e43736f737aae47c7b1a67ea57d6470a98d82e503ca9888237ba7 Jul 30 21:18:11.688187 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:11.688158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" event={"ID":"5d41e11a-b58e-4063-a477-9ea2ae50fdd2","Type":"ContainerStarted","Data":"738fea561e5e43736f737aae47c7b1a67ea57d6470a98d82e503ca9888237ba7"} Jul 30 21:18:12.696372 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:12.696282 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d41e11a-b58e-4063-a477-9ea2ae50fdd2" containerID="7da8e3b9da4538e21c128debe52af5f8c3455710f486a8c2115fa60fd8a49a45" exitCode=0 Jul 30 21:18:12.696372 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:12.696343 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" event={"ID":"5d41e11a-b58e-4063-a477-9ea2ae50fdd2","Type":"ContainerDied","Data":"7da8e3b9da4538e21c128debe52af5f8c3455710f486a8c2115fa60fd8a49a45"} Jul 30 21:18:13.849777 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:13.849750 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" Jul 30 21:18:13.993133 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:13.993022 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d41e11a-b58e-4063-a477-9ea2ae50fdd2-maas-api-service-ca\") pod \"5d41e11a-b58e-4063-a477-9ea2ae50fdd2\" (UID: \"5d41e11a-b58e-4063-a477-9ea2ae50fdd2\") " Jul 30 21:18:13.993398 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:13.993374 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d41e11a-b58e-4063-a477-9ea2ae50fdd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d41e11a-b58e-4063-a477-9ea2ae50fdd2" (UID: "5d41e11a-b58e-4063-a477-9ea2ae50fdd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:18:14.094148 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:14.094113 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d41e11a-b58e-4063-a477-9ea2ae50fdd2-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:14.708848 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:14.708812 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" event={"ID":"5d41e11a-b58e-4063-a477-9ea2ae50fdd2","Type":"ContainerDied","Data":"738fea561e5e43736f737aae47c7b1a67ea57d6470a98d82e503ca9888237ba7"} Jul 30 21:18:14.708848 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:14.708838 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5" Jul 30 21:18:14.709080 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:14.708871 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="738fea561e5e43736f737aae47c7b1a67ea57d6470a98d82e503ca9888237ba7" Jul 30 21:18:17.339612 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.339573 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx"] Jul 30 21:18:17.340088 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.339865 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" podUID="b9593de9-4c07-4017-a234-4c07cd461a3f" containerName="payload-processing" containerID="cri-o://3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6" gracePeriod=30 Jul 30 21:18:17.349097 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.349033 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8"] Jul 30 21:18:17.349402 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.349362 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" podUID="c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439" containerName="payload-pre-processing" containerID="cri-o://6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f" gracePeriod=30 Jul 30 21:18:17.485957 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.485918 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5"] Jul 30 21:18:17.489314 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.489275 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e026c1-vzzd5"] Jul 30 21:18:17.645601 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.645576 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:18:17.648336 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.648317 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:18:17.726533 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.726495 2575 generic.go:356] "Generic (PLEG): container finished" podID="c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439" containerID="6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f" exitCode=0 Jul 30 21:18:17.726734 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.726539 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" Jul 30 21:18:17.726734 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.726583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" event={"ID":"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439","Type":"ContainerDied","Data":"6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f"} Jul 30 21:18:17.726734 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.726641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8" event={"ID":"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439","Type":"ContainerDied","Data":"77d15873723e8ddf6cf08ac71639e44cb1706ec74c08fef29c4cc6949c19edfe"} Jul 30 21:18:17.726734 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.726667 2575 scope.go:117] "RemoveContainer" containerID="6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f" Jul 30 21:18:17.727881 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.727859 2575 generic.go:356] "Generic (PLEG): container finished" podID="b9593de9-4c07-4017-a234-4c07cd461a3f" containerID="3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6" exitCode=0 Jul 30 21:18:17.728015 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.727893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" event={"ID":"b9593de9-4c07-4017-a234-4c07cd461a3f","Type":"ContainerDied","Data":"3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6"} Jul 30 21:18:17.728015 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.727910 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" Jul 30 21:18:17.728015 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.727936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx" event={"ID":"b9593de9-4c07-4017-a234-4c07cd461a3f","Type":"ContainerDied","Data":"fb8485e21ca05f9edfbdce675144584bb522881dc1a60c917b4f360bb408dc11"} Jul 30 21:18:17.738047 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.738019 2575 scope.go:117] "RemoveContainer" containerID="6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f" Jul 30 21:18:17.738320 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:18:17.738301 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f\": container with ID starting with 6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f not found: ID does not exist" containerID="6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f" Jul 30 21:18:17.738382 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.738329 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f"} err="failed to get container status \"6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f\": rpc error: code = NotFound desc = could not find container \"6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f\": container with ID starting with 6b932a3685a02ceb8cac97fe21712339fc074c5bc2801262be5e5eaea506a50f not found: ID does not exist" Jul 30 21:18:17.738382 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.738347 2575 scope.go:117] "RemoveContainer" containerID="3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6" Jul 30 21:18:17.746462 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.746445 2575 scope.go:117] "RemoveContainer" containerID="3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6" Jul 30 21:18:17.746699 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:18:17.746678 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6\": container with ID starting with 3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6 not found: ID does not exist" containerID="3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6" Jul 30 21:18:17.746761 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.746704 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6"} err="failed to get container status \"3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6\": rpc error: code = NotFound desc = could not find container \"3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6\": container with ID starting with 3d84b41e399462022ed4119de038650b1f743e1ab79ce80a9991b9d059f30ff6 not found: ID does not exist" Jul 30 21:18:17.828677 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.828646 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-plugins-config-volume\") pod \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\" (UID: \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\") " Jul 30 21:18:17.828828 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.828722 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9593de9-4c07-4017-a234-4c07cd461a3f-plugins-config-volume\") pod \"b9593de9-4c07-4017-a234-4c07cd461a3f\" (UID: \"b9593de9-4c07-4017-a234-4c07cd461a3f\") " Jul 30 21:18:17.828828 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.828800 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgw4z\" (UniqueName: \"kubernetes.io/projected/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-kube-api-access-rgw4z\") pod \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\" (UID: \"c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439\") " Jul 30 21:18:17.828915 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.828860 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwfmj\" (UniqueName: \"kubernetes.io/projected/b9593de9-4c07-4017-a234-4c07cd461a3f-kube-api-access-kwfmj\") pod \"b9593de9-4c07-4017-a234-4c07cd461a3f\" (UID: \"b9593de9-4c07-4017-a234-4c07cd461a3f\") " Jul 30 21:18:17.829146 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.829119 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439" (UID: "c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:18:17.829245 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.829128 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9593de9-4c07-4017-a234-4c07cd461a3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9593de9-4c07-4017-a234-4c07cd461a3f" (UID: "b9593de9-4c07-4017-a234-4c07cd461a3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:18:17.830917 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.830893 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9593de9-4c07-4017-a234-4c07cd461a3f-kube-api-access-kwfmj" (OuterVolumeSpecName: "kube-api-access-kwfmj") pod "b9593de9-4c07-4017-a234-4c07cd461a3f" (UID: "b9593de9-4c07-4017-a234-4c07cd461a3f"). InnerVolumeSpecName "kube-api-access-kwfmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:18:17.830990 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.830924 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-kube-api-access-rgw4z" (OuterVolumeSpecName: "kube-api-access-rgw4z") pod "c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439" (UID: "c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439"). InnerVolumeSpecName "kube-api-access-rgw4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:18:17.929657 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.929564 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:17.929657 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.929595 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9593de9-4c07-4017-a234-4c07cd461a3f-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:17.929657 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.929608 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgw4z\" (UniqueName: \"kubernetes.io/projected/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439-kube-api-access-rgw4z\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:17.929657 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:17.929620 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwfmj\" (UniqueName: \"kubernetes.io/projected/b9593de9-4c07-4017-a234-4c07cd461a3f-kube-api-access-kwfmj\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:18.014201 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:18.014171 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d41e11a-b58e-4063-a477-9ea2ae50fdd2" path="/var/lib/kubelet/pods/5d41e11a-b58e-4063-a477-9ea2ae50fdd2/volumes" Jul 30 21:18:18.042691 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:18.042663 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8"] Jul 30 21:18:18.046358 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:18.046331 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e026c1-6b4c7449fb-qnjn8"] Jul 30 21:18:18.058869 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:18.058842 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx"] Jul 30 21:18:18.063816 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:18.063793 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e026c1-5557cf94db-vrbzx"] Jul 30 21:18:20.013659 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:20.013624 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9593de9-4c07-4017-a234-4c07cd461a3f" path="/var/lib/kubelet/pods/b9593de9-4c07-4017-a234-4c07cd461a3f/volumes" Jul 30 21:18:20.014049 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:20.014002 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439" path="/var/lib/kubelet/pods/c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439/volumes" Jul 30 21:18:27.695203 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695165 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn"] Jul 30 21:18:27.695647 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695633 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9593de9-4c07-4017-a234-4c07cd461a3f" containerName="payload-processing" Jul 30 21:18:27.695692 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695648 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9593de9-4c07-4017-a234-4c07cd461a3f" containerName="payload-processing" Jul 30 21:18:27.695692 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695664 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439" containerName="payload-pre-processing" Jul 30 21:18:27.695692 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695670 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439" containerName="payload-pre-processing" Jul 30 21:18:27.695692 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695677 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d41e11a-b58e-4063-a477-9ea2ae50fdd2" containerName="revoke-keys" Jul 30 21:18:27.695692 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695688 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d41e11a-b58e-4063-a477-9ea2ae50fdd2" containerName="revoke-keys" Jul 30 21:18:27.695886 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695753 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9593de9-4c07-4017-a234-4c07cd461a3f" containerName="payload-processing" Jul 30 21:18:27.695886 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695761 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d41e11a-b58e-4063-a477-9ea2ae50fdd2" containerName="revoke-keys" Jul 30 21:18:27.695886 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.695772 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73d4ff0-3a00-4eb5-b8e2-163a7b7b5439" containerName="payload-pre-processing" Jul 30 21:18:27.700521 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.700503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" Jul 30 21:18:27.703166 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.703133 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:18:27.705356 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.705332 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn"] Jul 30 21:18:27.709480 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.709171 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589b61c-893d-4b76-989e-be7e528874ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn\" (UID: \"9589b61c-893d-4b76-989e-be7e528874ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" Jul 30 21:18:27.810341 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.810298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589b61c-893d-4b76-989e-be7e528874ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn\" (UID: \"9589b61c-893d-4b76-989e-be7e528874ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" Jul 30 21:18:27.810887 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:27.810866 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589b61c-893d-4b76-989e-be7e528874ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn\" (UID: \"9589b61c-893d-4b76-989e-be7e528874ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" Jul 30 21:18:28.013583 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:28.013558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" Jul 30 21:18:28.156159 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:28.156125 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn"] Jul 30 21:18:28.157865 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:18:28.157822 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9589b61c_893d_4b76_989e_be7e528874ae.slice/crio-6e266035149f56616c3864b7fc0a41d5cd5b0367ade16c3100d252d6cffd9238 WatchSource:0}: Error finding container 6e266035149f56616c3864b7fc0a41d5cd5b0367ade16c3100d252d6cffd9238: Status 404 returned error can't find the container with id 6e266035149f56616c3864b7fc0a41d5cd5b0367ade16c3100d252d6cffd9238 Jul 30 21:18:28.801139 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:28.801097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" event={"ID":"9589b61c-893d-4b76-989e-be7e528874ae","Type":"ContainerStarted","Data":"b2df6171ce9a233f876a3d931aa5462d75ec2d3b10894a54f7a5f843d21d45c1"} Jul 30 21:18:28.801527 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:28.801165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" event={"ID":"9589b61c-893d-4b76-989e-be7e528874ae","Type":"ContainerStarted","Data":"6e266035149f56616c3864b7fc0a41d5cd5b0367ade16c3100d252d6cffd9238"} Jul 30 21:18:28.818834 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:28.818780 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" podStartSLOduration=1.818762416 podStartE2EDuration="1.818762416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:18:28.816380444 +0000 UTC m=+3447.281518794" watchObservedRunningTime="2026-07-30 21:18:28.818762416 +0000 UTC m=+3447.283900808" Jul 30 21:18:29.809312 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:29.809273 2575 generic.go:356] "Generic (PLEG): container finished" podID="9589b61c-893d-4b76-989e-be7e528874ae" containerID="b2df6171ce9a233f876a3d931aa5462d75ec2d3b10894a54f7a5f843d21d45c1" exitCode=0 Jul 30 21:18:29.809688 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:29.809352 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" event={"ID":"9589b61c-893d-4b76-989e-be7e528874ae","Type":"ContainerDied","Data":"b2df6171ce9a233f876a3d931aa5462d75ec2d3b10894a54f7a5f843d21d45c1"} Jul 30 21:18:30.960040 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:30.960014 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" Jul 30 21:18:31.032802 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:31.032766 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589b61c-893d-4b76-989e-be7e528874ae-maas-api-service-ca\") pod \"9589b61c-893d-4b76-989e-be7e528874ae\" (UID: \"9589b61c-893d-4b76-989e-be7e528874ae\") " Jul 30 21:18:31.033142 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:31.033120 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9589b61c-893d-4b76-989e-be7e528874ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9589b61c-893d-4b76-989e-be7e528874ae" (UID: "9589b61c-893d-4b76-989e-be7e528874ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:18:31.134272 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:31.134185 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589b61c-893d-4b76-989e-be7e528874ae-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:31.823362 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:31.823332 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" Jul 30 21:18:31.823362 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:31.823348 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn" event={"ID":"9589b61c-893d-4b76-989e-be7e528874ae","Type":"ContainerDied","Data":"6e266035149f56616c3864b7fc0a41d5cd5b0367ade16c3100d252d6cffd9238"} Jul 30 21:18:31.823625 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:31.823397 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e266035149f56616c3864b7fc0a41d5cd5b0367ade16c3100d252d6cffd9238" Jul 30 21:18:43.265119 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.265085 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh"] Jul 30 21:18:43.265697 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.265358 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" podUID="207c9c60-3520-4da2-8dc9-cc9ea1b29929" containerName="payload-processing" containerID="cri-o://962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98" gracePeriod=30 Jul 30 21:18:43.274736 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.274649 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b"] Jul 30 21:18:43.338198 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:18:43.338156 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207c9c60_3520_4da2_8dc9_cc9ea1b29929.slice/crio-conmon-962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98.scope\": RecentStats: unable to find data in memory cache]" Jul 30 21:18:43.412338 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.412271 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn"] Jul 30 21:18:43.415214 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.415180 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-720e63-hk6sn"] Jul 30 21:18:43.534006 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.533974 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:18:43.646273 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.646231 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k8lf\" (UniqueName: \"kubernetes.io/projected/207c9c60-3520-4da2-8dc9-cc9ea1b29929-kube-api-access-4k8lf\") pod \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\" (UID: \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\") " Jul 30 21:18:43.646482 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.646298 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c9c60-3520-4da2-8dc9-cc9ea1b29929-plugins-config-volume\") pod \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\" (UID: \"207c9c60-3520-4da2-8dc9-cc9ea1b29929\") " Jul 30 21:18:43.646734 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.646704 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207c9c60-3520-4da2-8dc9-cc9ea1b29929-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "207c9c60-3520-4da2-8dc9-cc9ea1b29929" (UID: "207c9c60-3520-4da2-8dc9-cc9ea1b29929"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:18:43.648453 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.648429 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207c9c60-3520-4da2-8dc9-cc9ea1b29929-kube-api-access-4k8lf" (OuterVolumeSpecName: "kube-api-access-4k8lf") pod "207c9c60-3520-4da2-8dc9-cc9ea1b29929" (UID: "207c9c60-3520-4da2-8dc9-cc9ea1b29929"). InnerVolumeSpecName "kube-api-access-4k8lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:18:43.747294 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.747257 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k8lf\" (UniqueName: \"kubernetes.io/projected/207c9c60-3520-4da2-8dc9-cc9ea1b29929-kube-api-access-4k8lf\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:43.747294 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.747287 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c9c60-3520-4da2-8dc9-cc9ea1b29929-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:43.898188 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.898101 2575 generic.go:356] "Generic (PLEG): container finished" podID="207c9c60-3520-4da2-8dc9-cc9ea1b29929" containerID="962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98" exitCode=0 Jul 30 21:18:43.898188 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.898149 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" Jul 30 21:18:43.898382 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.898191 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" event={"ID":"207c9c60-3520-4da2-8dc9-cc9ea1b29929","Type":"ContainerDied","Data":"962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98"} Jul 30 21:18:43.898382 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.898257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh" event={"ID":"207c9c60-3520-4da2-8dc9-cc9ea1b29929","Type":"ContainerDied","Data":"2e32d37bbec86ac9fff8e1da53db277b393715532d90d31b5feb44f3d8c1bea3"} Jul 30 21:18:43.898382 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.898291 2575 scope.go:117] "RemoveContainer" containerID="962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98" Jul 30 21:18:43.898530 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.898425 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" podUID="fa363f9c-f9aa-4379-85a9-d5466bea76fd" containerName="payload-pre-processing" containerID="cri-o://82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61" gracePeriod=30 Jul 30 21:18:43.908514 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.908493 2575 scope.go:117] "RemoveContainer" containerID="962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98" Jul 30 21:18:43.910137 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:18:43.908848 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98\": container with ID starting with 962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98 not found: ID does not exist" containerID="962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98" Jul 30 21:18:43.910137 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.909370 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98"} err="failed to get container status \"962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98\": rpc error: code = NotFound desc = could not find container \"962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98\": container with ID starting with 962f595e3fc83efb419377b74606490775da6451ee5781d24c62e24eb9bc7b98 not found: ID does not exist" Jul 30 21:18:43.935207 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.935170 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh"] Jul 30 21:18:43.942014 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:43.941985 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-720e63-578d67d54d-sg4kh"] Jul 30 21:18:44.015347 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.015314 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="207c9c60-3520-4da2-8dc9-cc9ea1b29929" path="/var/lib/kubelet/pods/207c9c60-3520-4da2-8dc9-cc9ea1b29929/volumes" Jul 30 21:18:44.015663 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.015646 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9589b61c-893d-4b76-989e-be7e528874ae" path="/var/lib/kubelet/pods/9589b61c-893d-4b76-989e-be7e528874ae/volumes" Jul 30 21:18:44.166240 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.166214 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:18:44.252721 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.252690 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvzjs\" (UniqueName: \"kubernetes.io/projected/fa363f9c-f9aa-4379-85a9-d5466bea76fd-kube-api-access-cvzjs\") pod \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\" (UID: \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\") " Jul 30 21:18:44.252905 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.252804 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa363f9c-f9aa-4379-85a9-d5466bea76fd-plugins-config-volume\") pod \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\" (UID: \"fa363f9c-f9aa-4379-85a9-d5466bea76fd\") " Jul 30 21:18:44.253200 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.253176 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa363f9c-f9aa-4379-85a9-d5466bea76fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa363f9c-f9aa-4379-85a9-d5466bea76fd" (UID: "fa363f9c-f9aa-4379-85a9-d5466bea76fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:18:44.254798 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.254779 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa363f9c-f9aa-4379-85a9-d5466bea76fd-kube-api-access-cvzjs" (OuterVolumeSpecName: "kube-api-access-cvzjs") pod "fa363f9c-f9aa-4379-85a9-d5466bea76fd" (UID: "fa363f9c-f9aa-4379-85a9-d5466bea76fd"). InnerVolumeSpecName "kube-api-access-cvzjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:18:44.354134 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.354049 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa363f9c-f9aa-4379-85a9-d5466bea76fd-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:44.354134 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.354134 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvzjs\" (UniqueName: \"kubernetes.io/projected/fa363f9c-f9aa-4379-85a9-d5466bea76fd-kube-api-access-cvzjs\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:18:44.906509 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.906474 2575 generic.go:356] "Generic (PLEG): container finished" podID="fa363f9c-f9aa-4379-85a9-d5466bea76fd" containerID="82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61" exitCode=0 Jul 30 21:18:44.906694 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.906517 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" event={"ID":"fa363f9c-f9aa-4379-85a9-d5466bea76fd","Type":"ContainerDied","Data":"82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61"} Jul 30 21:18:44.906694 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.906555 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" event={"ID":"fa363f9c-f9aa-4379-85a9-d5466bea76fd","Type":"ContainerDied","Data":"ef0a3e75531cbb5c082ef06bd82151ce47fdcdd8a496f0f31a84f6659fb6d78a"} Jul 30 21:18:44.906694 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.906576 2575 scope.go:117] "RemoveContainer" containerID="82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61" Jul 30 21:18:44.906694 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.906517 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b" Jul 30 21:18:44.916527 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.916508 2575 scope.go:117] "RemoveContainer" containerID="82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61" Jul 30 21:18:44.916769 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:18:44.916752 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61\": container with ID starting with 82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61 not found: ID does not exist" containerID="82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61" Jul 30 21:18:44.916816 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.916778 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61"} err="failed to get container status \"82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61\": rpc error: code = NotFound desc = could not find container \"82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61\": container with ID starting with 82e9dc4c72c4e85d08d8e3e4cca420c521fe0703da88189fc02239eb36aa4d61 not found: ID does not exist" Jul 30 21:18:44.929108 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.929076 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b"] Jul 30 21:18:44.932555 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:44.932533 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-720e63-6cc46c8c84-n7q9b"] Jul 30 21:18:46.013280 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:18:46.013247 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa363f9c-f9aa-4379-85a9-d5466bea76fd" path="/var/lib/kubelet/pods/fa363f9c-f9aa-4379-85a9-d5466bea76fd/volumes" Jul 30 21:19:37.929397 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929359 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5"] Jul 30 21:19:37.929829 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929789 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa363f9c-f9aa-4379-85a9-d5466bea76fd" containerName="payload-pre-processing" Jul 30 21:19:37.929829 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929802 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa363f9c-f9aa-4379-85a9-d5466bea76fd" containerName="payload-pre-processing" Jul 30 21:19:37.929829 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929812 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="207c9c60-3520-4da2-8dc9-cc9ea1b29929" containerName="payload-processing" Jul 30 21:19:37.929829 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929818 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="207c9c60-3520-4da2-8dc9-cc9ea1b29929" containerName="payload-processing" Jul 30 21:19:37.929829 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929827 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9589b61c-893d-4b76-989e-be7e528874ae" containerName="revoke-keys" Jul 30 21:19:37.929829 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929833 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9589b61c-893d-4b76-989e-be7e528874ae" containerName="revoke-keys" Jul 30 21:19:37.930034 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929904 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9589b61c-893d-4b76-989e-be7e528874ae" containerName="revoke-keys" Jul 30 21:19:37.930034 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929912 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa363f9c-f9aa-4379-85a9-d5466bea76fd" containerName="payload-pre-processing" Jul 30 21:19:37.930034 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.929917 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="207c9c60-3520-4da2-8dc9-cc9ea1b29929" containerName="payload-processing" Jul 30 21:19:37.933141 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.933124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" Jul 30 21:19:37.935969 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.935946 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:19:37.945860 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:37.945836 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5"] Jul 30 21:19:38.026698 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:38.026667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357a2408-9d10-41cd-ab0c-b636652d6970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5\" (UID: \"357a2408-9d10-41cd-ab0c-b636652d6970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" Jul 30 21:19:38.127337 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:38.127299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357a2408-9d10-41cd-ab0c-b636652d6970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5\" (UID: \"357a2408-9d10-41cd-ab0c-b636652d6970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" Jul 30 21:19:38.128031 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:38.128007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357a2408-9d10-41cd-ab0c-b636652d6970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5\" (UID: \"357a2408-9d10-41cd-ab0c-b636652d6970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" Jul 30 21:19:38.243783 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:38.243751 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" Jul 30 21:19:38.394031 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:38.393995 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5"] Jul 30 21:19:38.394493 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:19:38.394469 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357a2408_9d10_41cd_ab0c_b636652d6970.slice/crio-6d094fff4a64264dab7a84d29a053e18eb94bc941f46e9679a25fafbb2738900 WatchSource:0}: Error finding container 6d094fff4a64264dab7a84d29a053e18eb94bc941f46e9679a25fafbb2738900: Status 404 returned error can't find the container with id 6d094fff4a64264dab7a84d29a053e18eb94bc941f46e9679a25fafbb2738900 Jul 30 21:19:39.219733 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:39.219693 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" event={"ID":"357a2408-9d10-41cd-ab0c-b636652d6970","Type":"ContainerStarted","Data":"241cc04c25e01b4f828dcdc530c04acdc88391577250b41e61d2f44451ccc0dc"} Jul 30 21:19:39.220158 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:39.219758 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" event={"ID":"357a2408-9d10-41cd-ab0c-b636652d6970","Type":"ContainerStarted","Data":"6d094fff4a64264dab7a84d29a053e18eb94bc941f46e9679a25fafbb2738900"} Jul 30 21:19:39.236500 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:39.236456 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" podStartSLOduration=2.236438744 podStartE2EDuration="2.236438744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:19:39.234679308 +0000 UTC m=+3517.699817656" watchObservedRunningTime="2026-07-30 21:19:39.236438744 +0000 UTC m=+3517.701577095" Jul 30 21:19:40.227196 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:40.227164 2575 generic.go:356] "Generic (PLEG): container finished" podID="357a2408-9d10-41cd-ab0c-b636652d6970" containerID="241cc04c25e01b4f828dcdc530c04acdc88391577250b41e61d2f44451ccc0dc" exitCode=0 Jul 30 21:19:40.227661 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:40.227255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" event={"ID":"357a2408-9d10-41cd-ab0c-b636652d6970","Type":"ContainerDied","Data":"241cc04c25e01b4f828dcdc530c04acdc88391577250b41e61d2f44451ccc0dc"} Jul 30 21:19:41.369207 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:41.369180 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" Jul 30 21:19:41.457317 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:41.457277 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357a2408-9d10-41cd-ab0c-b636652d6970-maas-api-service-ca\") pod \"357a2408-9d10-41cd-ab0c-b636652d6970\" (UID: \"357a2408-9d10-41cd-ab0c-b636652d6970\") " Jul 30 21:19:41.457648 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:41.457625 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357a2408-9d10-41cd-ab0c-b636652d6970-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "357a2408-9d10-41cd-ab0c-b636652d6970" (UID: "357a2408-9d10-41cd-ab0c-b636652d6970"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:19:41.557992 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:41.557909 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357a2408-9d10-41cd-ab0c-b636652d6970-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:19:42.240364 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:42.240336 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" Jul 30 21:19:42.240532 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:42.240375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5" event={"ID":"357a2408-9d10-41cd-ab0c-b636652d6970","Type":"ContainerDied","Data":"6d094fff4a64264dab7a84d29a053e18eb94bc941f46e9679a25fafbb2738900"} Jul 30 21:19:42.240532 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:42.240421 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d094fff4a64264dab7a84d29a053e18eb94bc941f46e9679a25fafbb2738900" Jul 30 21:19:48.539757 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.539721 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x"] Jul 30 21:19:48.540591 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.540540 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" podUID="9530be12-a2d2-41f8-a20c-1801ec6aab61" containerName="payload-processing" containerID="cri-o://f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01" gracePeriod=30 Jul 30 21:19:48.545930 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.545903 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4"] Jul 30 21:19:48.546204 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.546173 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" podUID="872d4680-b538-48c7-bb03-d63acafcb058" containerName="payload-pre-processing" containerID="cri-o://17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405" gracePeriod=30 Jul 30 21:19:48.738048 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.738009 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5"] Jul 30 21:19:48.741246 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.741222 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6967f3-m2fn5"] Jul 30 21:19:48.821653 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.821624 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:19:48.851914 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.851888 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:19:48.926012 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.925975 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872d4680-b538-48c7-bb03-d63acafcb058-plugins-config-volume\") pod \"872d4680-b538-48c7-bb03-d63acafcb058\" (UID: \"872d4680-b538-48c7-bb03-d63acafcb058\") " Jul 30 21:19:48.926243 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.926050 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9530be12-a2d2-41f8-a20c-1801ec6aab61-plugins-config-volume\") pod \"9530be12-a2d2-41f8-a20c-1801ec6aab61\" (UID: \"9530be12-a2d2-41f8-a20c-1801ec6aab61\") " Jul 30 21:19:48.926243 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.926105 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbq2t\" (UniqueName: \"kubernetes.io/projected/872d4680-b538-48c7-bb03-d63acafcb058-kube-api-access-gbq2t\") pod \"872d4680-b538-48c7-bb03-d63acafcb058\" (UID: \"872d4680-b538-48c7-bb03-d63acafcb058\") " Jul 30 21:19:48.926243 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.926155 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7489t\" (UniqueName: \"kubernetes.io/projected/9530be12-a2d2-41f8-a20c-1801ec6aab61-kube-api-access-7489t\") pod \"9530be12-a2d2-41f8-a20c-1801ec6aab61\" (UID: \"9530be12-a2d2-41f8-a20c-1801ec6aab61\") " Jul 30 21:19:48.926499 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.926466 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9530be12-a2d2-41f8-a20c-1801ec6aab61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9530be12-a2d2-41f8-a20c-1801ec6aab61" (UID: "9530be12-a2d2-41f8-a20c-1801ec6aab61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:19:48.926572 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.926538 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872d4680-b538-48c7-bb03-d63acafcb058-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "872d4680-b538-48c7-bb03-d63acafcb058" (UID: "872d4680-b538-48c7-bb03-d63acafcb058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:19:48.928245 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.928219 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872d4680-b538-48c7-bb03-d63acafcb058-kube-api-access-gbq2t" (OuterVolumeSpecName: "kube-api-access-gbq2t") pod "872d4680-b538-48c7-bb03-d63acafcb058" (UID: "872d4680-b538-48c7-bb03-d63acafcb058"). InnerVolumeSpecName "kube-api-access-gbq2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:19:48.928309 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:48.928259 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9530be12-a2d2-41f8-a20c-1801ec6aab61-kube-api-access-7489t" (OuterVolumeSpecName: "kube-api-access-7489t") pod "9530be12-a2d2-41f8-a20c-1801ec6aab61" (UID: "9530be12-a2d2-41f8-a20c-1801ec6aab61"). InnerVolumeSpecName "kube-api-access-7489t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:19:49.027157 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.027120 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9530be12-a2d2-41f8-a20c-1801ec6aab61-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:19:49.027157 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.027149 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbq2t\" (UniqueName: \"kubernetes.io/projected/872d4680-b538-48c7-bb03-d63acafcb058-kube-api-access-gbq2t\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:19:49.027157 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.027160 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7489t\" (UniqueName: \"kubernetes.io/projected/9530be12-a2d2-41f8-a20c-1801ec6aab61-kube-api-access-7489t\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:19:49.027157 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.027168 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872d4680-b538-48c7-bb03-d63acafcb058-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:19:49.280417 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.280383 2575 generic.go:356] "Generic (PLEG): container finished" podID="872d4680-b538-48c7-bb03-d63acafcb058" containerID="17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405" exitCode=0 Jul 30 21:19:49.280619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.280434 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" Jul 30 21:19:49.280619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.280471 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" event={"ID":"872d4680-b538-48c7-bb03-d63acafcb058","Type":"ContainerDied","Data":"17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405"} Jul 30 21:19:49.280619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.280530 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4" event={"ID":"872d4680-b538-48c7-bb03-d63acafcb058","Type":"ContainerDied","Data":"6a4e6e308242aac502ab9d4ea5deb01973c15498381778e39d1f0e69ed24f9da"} Jul 30 21:19:49.280619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.280565 2575 scope.go:117] "RemoveContainer" containerID="17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405" Jul 30 21:19:49.281786 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.281767 2575 generic.go:356] "Generic (PLEG): container finished" podID="9530be12-a2d2-41f8-a20c-1801ec6aab61" containerID="f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01" exitCode=0 Jul 30 21:19:49.281872 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.281804 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" Jul 30 21:19:49.281872 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.281838 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" event={"ID":"9530be12-a2d2-41f8-a20c-1801ec6aab61","Type":"ContainerDied","Data":"f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01"} Jul 30 21:19:49.281964 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.281885 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x" event={"ID":"9530be12-a2d2-41f8-a20c-1801ec6aab61","Type":"ContainerDied","Data":"0e40432d11c6393162015827a9e0ed55fa241c39b324803dbeb9497a20d9b3cf"} Jul 30 21:19:49.291501 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.291482 2575 scope.go:117] "RemoveContainer" containerID="17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405" Jul 30 21:19:49.291803 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:19:49.291783 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405\": container with ID starting with 17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405 not found: ID does not exist" containerID="17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405" Jul 30 21:19:49.291889 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.291815 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405"} err="failed to get container status \"17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405\": rpc error: code = NotFound desc = could not find container \"17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405\": container with ID starting with 17247925970f367bad80c8ba06871846a4a8f6f9be648ea50f9bd19150064405 not found: ID does not exist" Jul 30 21:19:49.291889 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.291839 2575 scope.go:117] "RemoveContainer" containerID="f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01" Jul 30 21:19:49.300604 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.300586 2575 scope.go:117] "RemoveContainer" containerID="f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01" Jul 30 21:19:49.300880 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:19:49.300862 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01\": container with ID starting with f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01 not found: ID does not exist" containerID="f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01" Jul 30 21:19:49.300952 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.300893 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01"} err="failed to get container status \"f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01\": rpc error: code = NotFound desc = could not find container \"f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01\": container with ID starting with f66330246c8d7544f70c6732b8ac6b6f5868d43580566863b8677e81dc2cad01 not found: ID does not exist" Jul 30 21:19:49.307867 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.307842 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4"] Jul 30 21:19:49.313955 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.313931 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6967f3-66d7d7c4d4-26ls4"] Jul 30 21:19:49.325528 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.325504 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x"] Jul 30 21:19:49.330150 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:49.330128 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6967f3-d5d886cdb-hwg5x"] Jul 30 21:19:50.013209 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:50.013175 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357a2408-9d10-41cd-ab0c-b636652d6970" path="/var/lib/kubelet/pods/357a2408-9d10-41cd-ab0c-b636652d6970/volumes" Jul 30 21:19:50.013582 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:50.013553 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872d4680-b538-48c7-bb03-d63acafcb058" path="/var/lib/kubelet/pods/872d4680-b538-48c7-bb03-d63acafcb058/volumes" Jul 30 21:19:50.013886 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:50.013871 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9530be12-a2d2-41f8-a20c-1801ec6aab61" path="/var/lib/kubelet/pods/9530be12-a2d2-41f8-a20c-1801ec6aab61/volumes" Jul 30 21:19:55.515762 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.515722 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm"] Jul 30 21:19:55.516342 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516322 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872d4680-b538-48c7-bb03-d63acafcb058" containerName="payload-pre-processing" Jul 30 21:19:55.516399 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516346 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="872d4680-b538-48c7-bb03-d63acafcb058" containerName="payload-pre-processing" Jul 30 21:19:55.516399 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516387 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9530be12-a2d2-41f8-a20c-1801ec6aab61" containerName="payload-processing" Jul 30 21:19:55.516462 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516397 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9530be12-a2d2-41f8-a20c-1801ec6aab61" containerName="payload-processing" Jul 30 21:19:55.516462 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516423 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357a2408-9d10-41cd-ab0c-b636652d6970" containerName="revoke-keys" Jul 30 21:19:55.516462 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516431 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="357a2408-9d10-41cd-ab0c-b636652d6970" containerName="revoke-keys" Jul 30 21:19:55.516551 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516533 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9530be12-a2d2-41f8-a20c-1801ec6aab61" containerName="payload-processing" Jul 30 21:19:55.516551 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516548 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="872d4680-b538-48c7-bb03-d63acafcb058" containerName="payload-pre-processing" Jul 30 21:19:55.516620 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.516560 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="357a2408-9d10-41cd-ab0c-b636652d6970" containerName="revoke-keys" Jul 30 21:19:55.521633 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.521602 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" Jul 30 21:19:55.524626 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.524605 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:19:55.534394 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.534357 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm"] Jul 30 21:19:55.583220 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.583178 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm\" (UID: \"7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" Jul 30 21:19:55.684071 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.684029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm\" (UID: \"7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" Jul 30 21:19:55.684688 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.684668 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm\" (UID: \"7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" Jul 30 21:19:55.834889 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.834800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" Jul 30 21:19:55.983304 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:55.983277 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm"] Jul 30 21:19:55.985130 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:19:55.985101 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa67e01_7d14_4c5e_8e63_5cf85b81cbf9.slice/crio-411d042e177297f8e0498ff053e9dcc02acdaab4ae69b6fb0c383afd139f8190 WatchSource:0}: Error finding container 411d042e177297f8e0498ff053e9dcc02acdaab4ae69b6fb0c383afd139f8190: Status 404 returned error can't find the container with id 411d042e177297f8e0498ff053e9dcc02acdaab4ae69b6fb0c383afd139f8190 Jul 30 21:19:56.328820 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:56.328787 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" event={"ID":"7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9","Type":"ContainerStarted","Data":"411d042e177297f8e0498ff053e9dcc02acdaab4ae69b6fb0c383afd139f8190"} Jul 30 21:19:57.340328 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:57.340287 2575 generic.go:356] "Generic (PLEG): container finished" podID="7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9" containerID="f4cf04cb5595c2f5ab53536f89d2ec12e72d09fab275d8ccab7025562a83baba" exitCode=0 Jul 30 21:19:57.340792 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:57.340381 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" event={"ID":"7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9","Type":"ContainerDied","Data":"f4cf04cb5595c2f5ab53536f89d2ec12e72d09fab275d8ccab7025562a83baba"} Jul 30 21:19:58.491113 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:58.491081 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" Jul 30 21:19:58.610484 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:58.610385 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9-maas-api-service-ca\") pod \"7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9\" (UID: \"7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9\") " Jul 30 21:19:58.610781 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:58.610757 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9" (UID: "7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:19:58.711729 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:58.711688 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:19:59.353772 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:59.353734 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" event={"ID":"7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9","Type":"ContainerDied","Data":"411d042e177297f8e0498ff053e9dcc02acdaab4ae69b6fb0c383afd139f8190"} Jul 30 21:19:59.353930 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:59.353785 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="411d042e177297f8e0498ff053e9dcc02acdaab4ae69b6fb0c383afd139f8190" Jul 30 21:19:59.353930 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:19:59.353752 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm" Jul 30 21:20:05.939615 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:05.939583 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5"] Jul 30 21:20:05.941974 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:05.941945 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-55df2b-29757435-jnnv5"] Jul 30 21:20:05.970312 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:05.970276 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f"] Jul 30 21:20:05.970564 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:05.970538 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" podUID="08435e1f-ac77-4181-a9b0-18a8d4b9f984" containerName="payload-processing" containerID="cri-o://43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e" gracePeriod=30 Jul 30 21:20:05.981395 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:05.981349 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l"] Jul 30 21:20:05.981837 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:05.981642 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" podUID="a2e3862b-2010-49f5-af97-d9b4cc8c88b5" containerName="payload-pre-processing" containerID="cri-o://46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447" gracePeriod=30 Jul 30 21:20:06.017575 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.017540 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49514f54-b1eb-42d2-95dd-897da46ff7af" path="/var/lib/kubelet/pods/49514f54-b1eb-42d2-95dd-897da46ff7af/volumes" Jul 30 21:20:06.107231 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.107190 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm"] Jul 30 21:20:06.110091 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.110044 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-55df2b-ljjwm"] Jul 30 21:20:06.262605 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.262579 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:20:06.265521 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.265496 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:20:06.386162 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.386124 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-979lm\" (UniqueName: \"kubernetes.io/projected/08435e1f-ac77-4181-a9b0-18a8d4b9f984-kube-api-access-979lm\") pod \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\" (UID: \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\") " Jul 30 21:20:06.386346 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.386234 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srdf8\" (UniqueName: \"kubernetes.io/projected/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-kube-api-access-srdf8\") pod \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\" (UID: \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\") " Jul 30 21:20:06.386346 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.386294 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-plugins-config-volume\") pod \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\" (UID: \"a2e3862b-2010-49f5-af97-d9b4cc8c88b5\") " Jul 30 21:20:06.386346 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.386327 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08435e1f-ac77-4181-a9b0-18a8d4b9f984-plugins-config-volume\") pod \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\" (UID: \"08435e1f-ac77-4181-a9b0-18a8d4b9f984\") " Jul 30 21:20:06.386730 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.386701 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2e3862b-2010-49f5-af97-d9b4cc8c88b5" (UID: "a2e3862b-2010-49f5-af97-d9b4cc8c88b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:20:06.386813 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.386730 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08435e1f-ac77-4181-a9b0-18a8d4b9f984-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08435e1f-ac77-4181-a9b0-18a8d4b9f984" (UID: "08435e1f-ac77-4181-a9b0-18a8d4b9f984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:20:06.388465 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.388433 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08435e1f-ac77-4181-a9b0-18a8d4b9f984-kube-api-access-979lm" (OuterVolumeSpecName: "kube-api-access-979lm") pod "08435e1f-ac77-4181-a9b0-18a8d4b9f984" (UID: "08435e1f-ac77-4181-a9b0-18a8d4b9f984"). InnerVolumeSpecName "kube-api-access-979lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:20:06.388465 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.388461 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-kube-api-access-srdf8" (OuterVolumeSpecName: "kube-api-access-srdf8") pod "a2e3862b-2010-49f5-af97-d9b4cc8c88b5" (UID: "a2e3862b-2010-49f5-af97-d9b4cc8c88b5"). InnerVolumeSpecName "kube-api-access-srdf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:20:06.394527 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.394501 2575 generic.go:356] "Generic (PLEG): container finished" podID="08435e1f-ac77-4181-a9b0-18a8d4b9f984" containerID="43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e" exitCode=0 Jul 30 21:20:06.394638 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.394544 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" Jul 30 21:20:06.394638 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.394582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" event={"ID":"08435e1f-ac77-4181-a9b0-18a8d4b9f984","Type":"ContainerDied","Data":"43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e"} Jul 30 21:20:06.394721 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.394636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f" event={"ID":"08435e1f-ac77-4181-a9b0-18a8d4b9f984","Type":"ContainerDied","Data":"ffb40657583a430f95c13717e6c79afc97cf08a782786a24dfa200f490dec55f"} Jul 30 21:20:06.394721 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.394665 2575 scope.go:117] "RemoveContainer" containerID="43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e" Jul 30 21:20:06.395978 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.395958 2575 generic.go:356] "Generic (PLEG): container finished" podID="a2e3862b-2010-49f5-af97-d9b4cc8c88b5" containerID="46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447" exitCode=0 Jul 30 21:20:06.396068 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.395985 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" event={"ID":"a2e3862b-2010-49f5-af97-d9b4cc8c88b5","Type":"ContainerDied","Data":"46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447"} Jul 30 21:20:06.396068 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.396004 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" Jul 30 21:20:06.396068 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.396036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l" event={"ID":"a2e3862b-2010-49f5-af97-d9b4cc8c88b5","Type":"ContainerDied","Data":"626af5180e13f1af8db0e2e9795c6c27fe5bd9951f19190f3ff751754e5e951b"} Jul 30 21:20:06.404471 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.404448 2575 scope.go:117] "RemoveContainer" containerID="43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e" Jul 30 21:20:06.404783 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:20:06.404764 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e\": container with ID starting with 43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e not found: ID does not exist" containerID="43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e" Jul 30 21:20:06.404869 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.404797 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e"} err="failed to get container status \"43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e\": rpc error: code = NotFound desc = could not find container \"43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e\": container with ID starting with 43eeb91e1cf0d5ccec27768a35b60c4e37ce172a42026f0eb73140da5ae6d43e not found: ID does not exist" Jul 30 21:20:06.404869 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.404816 2575 scope.go:117] "RemoveContainer" containerID="46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447" Jul 30 21:20:06.413795 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.413776 2575 scope.go:117] "RemoveContainer" containerID="46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447" Jul 30 21:20:06.414095 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:20:06.414066 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447\": container with ID starting with 46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447 not found: ID does not exist" containerID="46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447" Jul 30 21:20:06.414200 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.414103 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447"} err="failed to get container status \"46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447\": rpc error: code = NotFound desc = could not find container \"46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447\": container with ID starting with 46b9562ac5d09e397f1c4531da4d64578f6ecbc81c6a06c0c71a1de5caa80447 not found: ID does not exist" Jul 30 21:20:06.422864 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.422836 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l"] Jul 30 21:20:06.426200 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.426172 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-55df2b-5cddcc4556-dls8l"] Jul 30 21:20:06.436533 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.436505 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f"] Jul 30 21:20:06.440377 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.440353 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-55df2b-7cb68799bb-92z7f"] Jul 30 21:20:06.487126 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.487091 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:06.487126 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.487121 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08435e1f-ac77-4181-a9b0-18a8d4b9f984-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:06.487126 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.487135 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-979lm\" (UniqueName: \"kubernetes.io/projected/08435e1f-ac77-4181-a9b0-18a8d4b9f984-kube-api-access-979lm\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:06.487374 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:06.487144 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srdf8\" (UniqueName: \"kubernetes.io/projected/a2e3862b-2010-49f5-af97-d9b4cc8c88b5-kube-api-access-srdf8\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:08.019075 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:08.013952 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08435e1f-ac77-4181-a9b0-18a8d4b9f984" path="/var/lib/kubelet/pods/08435e1f-ac77-4181-a9b0-18a8d4b9f984/volumes" Jul 30 21:20:08.019075 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:08.014537 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9" path="/var/lib/kubelet/pods/7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9/volumes" Jul 30 21:20:08.019075 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:08.015047 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e3862b-2010-49f5-af97-d9b4cc8c88b5" path="/var/lib/kubelet/pods/a2e3862b-2010-49f5-af97-d9b4cc8c88b5/volumes" Jul 30 21:20:13.322666 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.322632 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8"] Jul 30 21:20:13.323034 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323020 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9" containerName="revoke-keys" Jul 30 21:20:13.323108 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323036 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9" containerName="revoke-keys" Jul 30 21:20:13.323108 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323070 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e3862b-2010-49f5-af97-d9b4cc8c88b5" containerName="payload-pre-processing" Jul 30 21:20:13.323108 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323079 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e3862b-2010-49f5-af97-d9b4cc8c88b5" containerName="payload-pre-processing" Jul 30 21:20:13.323108 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323090 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08435e1f-ac77-4181-a9b0-18a8d4b9f984" containerName="payload-processing" Jul 30 21:20:13.323108 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323098 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="08435e1f-ac77-4181-a9b0-18a8d4b9f984" containerName="payload-processing" Jul 30 21:20:13.323265 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323164 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e3862b-2010-49f5-af97-d9b4cc8c88b5" containerName="payload-pre-processing" Jul 30 21:20:13.323265 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323176 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa67e01-7d14-4c5e-8e63-5cf85b81cbf9" containerName="revoke-keys" Jul 30 21:20:13.323265 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.323185 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="08435e1f-ac77-4181-a9b0-18a8d4b9f984" containerName="payload-processing" Jul 30 21:20:13.327551 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.327532 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" Jul 30 21:20:13.330677 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.330657 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-58j97\"" Jul 30 21:20:13.339429 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.339400 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8"] Jul 30 21:20:13.453508 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.453469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c08dcbb-8299-4d13-b23e-3fea12f49117-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8\" (UID: \"3c08dcbb-8299-4d13-b23e-3fea12f49117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" Jul 30 21:20:13.554014 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.553976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c08dcbb-8299-4d13-b23e-3fea12f49117-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8\" (UID: \"3c08dcbb-8299-4d13-b23e-3fea12f49117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" Jul 30 21:20:13.554619 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.554602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c08dcbb-8299-4d13-b23e-3fea12f49117-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8\" (UID: \"3c08dcbb-8299-4d13-b23e-3fea12f49117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" Jul 30 21:20:13.639214 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.639122 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" Jul 30 21:20:13.788862 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:13.788829 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8"] Jul 30 21:20:13.791645 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:20:13.791617 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c08dcbb_8299_4d13_b23e_3fea12f49117.slice/crio-f21a99750f54ef949171f8b4bb2c234b43d60a93f4cb1de330d7d57f02d9c983 WatchSource:0}: Error finding container f21a99750f54ef949171f8b4bb2c234b43d60a93f4cb1de330d7d57f02d9c983: Status 404 returned error can't find the container with id f21a99750f54ef949171f8b4bb2c234b43d60a93f4cb1de330d7d57f02d9c983 Jul 30 21:20:14.443827 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:14.443715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" event={"ID":"3c08dcbb-8299-4d13-b23e-3fea12f49117","Type":"ContainerStarted","Data":"52ab7c4f62e445af9c721074436c65715b19015f2978bf997fb191d6b065bd4c"} Jul 30 21:20:14.443827 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:14.443819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" event={"ID":"3c08dcbb-8299-4d13-b23e-3fea12f49117","Type":"ContainerStarted","Data":"f21a99750f54ef949171f8b4bb2c234b43d60a93f4cb1de330d7d57f02d9c983"} Jul 30 21:20:14.458959 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:14.458905 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" podStartSLOduration=1.458888973 podStartE2EDuration="1.458888973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:20:14.457807168 +0000 UTC m=+3552.922945516" watchObservedRunningTime="2026-07-30 21:20:14.458888973 +0000 UTC m=+3552.924027320" Jul 30 21:20:15.451769 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:15.451731 2575 generic.go:356] "Generic (PLEG): container finished" podID="3c08dcbb-8299-4d13-b23e-3fea12f49117" containerID="52ab7c4f62e445af9c721074436c65715b19015f2978bf997fb191d6b065bd4c" exitCode=0 Jul 30 21:20:15.452210 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:15.451789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" event={"ID":"3c08dcbb-8299-4d13-b23e-3fea12f49117","Type":"ContainerDied","Data":"52ab7c4f62e445af9c721074436c65715b19015f2978bf997fb191d6b065bd4c"} Jul 30 21:20:16.592919 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:16.592892 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" Jul 30 21:20:16.679918 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:16.679884 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c08dcbb-8299-4d13-b23e-3fea12f49117-maas-api-service-ca\") pod \"3c08dcbb-8299-4d13-b23e-3fea12f49117\" (UID: \"3c08dcbb-8299-4d13-b23e-3fea12f49117\") " Jul 30 21:20:16.680354 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:16.680329 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c08dcbb-8299-4d13-b23e-3fea12f49117-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c08dcbb-8299-4d13-b23e-3fea12f49117" (UID: "3c08dcbb-8299-4d13-b23e-3fea12f49117"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:20:16.780944 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:16.780850 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c08dcbb-8299-4d13-b23e-3fea12f49117-maas-api-service-ca\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:17.464685 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:17.464656 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" Jul 30 21:20:17.464858 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:17.464649 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8" event={"ID":"3c08dcbb-8299-4d13-b23e-3fea12f49117","Type":"ContainerDied","Data":"f21a99750f54ef949171f8b4bb2c234b43d60a93f4cb1de330d7d57f02d9c983"} Jul 30 21:20:17.464858 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:17.464789 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f21a99750f54ef949171f8b4bb2c234b43d60a93f4cb1de330d7d57f02d9c983" Jul 30 21:20:23.825300 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:23.825256 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf"] Jul 30 21:20:23.828174 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:23.828143 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e226df-29757435-9fgnf"] Jul 30 21:20:23.859121 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:23.859087 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp"] Jul 30 21:20:23.859361 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:23.859317 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" podUID="de003a9f-19c2-42c3-93ba-a97b658185cb" containerName="payload-processing" containerID="cri-o://e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344" gracePeriod=30 Jul 30 21:20:23.870170 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:23.870142 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm"] Jul 30 21:20:23.870396 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:23.870372 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" podUID="e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae" containerName="payload-pre-processing" containerID="cri-o://ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f" gracePeriod=30 Jul 30 21:20:24.005261 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.005205 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8"] Jul 30 21:20:24.007476 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.007448 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e226df-fhqf8"] Jul 30 21:20:24.015015 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.014985 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1686c1e9-7d74-4bef-96ba-8bdb2043baf7" path="/var/lib/kubelet/pods/1686c1e9-7d74-4bef-96ba-8bdb2043baf7/volumes" Jul 30 21:20:24.015358 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.015344 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c08dcbb-8299-4d13-b23e-3fea12f49117" path="/var/lib/kubelet/pods/3c08dcbb-8299-4d13-b23e-3fea12f49117/volumes" Jul 30 21:20:24.122792 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.122767 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:20:24.155306 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.155280 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:20:24.248183 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.248146 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5qlf\" (UniqueName: \"kubernetes.io/projected/de003a9f-19c2-42c3-93ba-a97b658185cb-kube-api-access-d5qlf\") pod \"de003a9f-19c2-42c3-93ba-a97b658185cb\" (UID: \"de003a9f-19c2-42c3-93ba-a97b658185cb\") " Jul 30 21:20:24.248390 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.248205 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de003a9f-19c2-42c3-93ba-a97b658185cb-plugins-config-volume\") pod \"de003a9f-19c2-42c3-93ba-a97b658185cb\" (UID: \"de003a9f-19c2-42c3-93ba-a97b658185cb\") " Jul 30 21:20:24.248611 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.248586 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de003a9f-19c2-42c3-93ba-a97b658185cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de003a9f-19c2-42c3-93ba-a97b658185cb" (UID: "de003a9f-19c2-42c3-93ba-a97b658185cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:20:24.250454 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.250430 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de003a9f-19c2-42c3-93ba-a97b658185cb-kube-api-access-d5qlf" (OuterVolumeSpecName: "kube-api-access-d5qlf") pod "de003a9f-19c2-42c3-93ba-a97b658185cb" (UID: "de003a9f-19c2-42c3-93ba-a97b658185cb"). InnerVolumeSpecName "kube-api-access-d5qlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:20:24.349570 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.349473 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkfpb\" (UniqueName: \"kubernetes.io/projected/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-kube-api-access-hkfpb\") pod \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\" (UID: \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\") " Jul 30 21:20:24.349750 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.349598 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-plugins-config-volume\") pod \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\" (UID: \"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae\") " Jul 30 21:20:24.349951 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.349930 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qlf\" (UniqueName: \"kubernetes.io/projected/de003a9f-19c2-42c3-93ba-a97b658185cb-kube-api-access-d5qlf\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:24.350025 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.349954 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de003a9f-19c2-42c3-93ba-a97b658185cb-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:24.350025 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.349991 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae" (UID: "e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:20:24.351583 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.351560 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-kube-api-access-hkfpb" (OuterVolumeSpecName: "kube-api-access-hkfpb") pod "e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae" (UID: "e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae"). InnerVolumeSpecName "kube-api-access-hkfpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:20:24.451405 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.451357 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-plugins-config-volume\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:24.451405 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.451402 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkfpb\" (UniqueName: \"kubernetes.io/projected/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae-kube-api-access-hkfpb\") on node \"ip-10-0-140-21.ec2.internal\" DevicePath \"\"" Jul 30 21:20:24.506297 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.506256 2575 generic.go:356] "Generic (PLEG): container finished" podID="de003a9f-19c2-42c3-93ba-a97b658185cb" containerID="e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344" exitCode=0 Jul 30 21:20:24.506297 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.506297 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" Jul 30 21:20:24.506542 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.506347 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" event={"ID":"de003a9f-19c2-42c3-93ba-a97b658185cb","Type":"ContainerDied","Data":"e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344"} Jul 30 21:20:24.506542 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.506401 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp" event={"ID":"de003a9f-19c2-42c3-93ba-a97b658185cb","Type":"ContainerDied","Data":"014b27365fd9204fceecd86b076f3d1cabc25555cb220eb87d9e6b0950d85a1c"} Jul 30 21:20:24.506542 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.506428 2575 scope.go:117] "RemoveContainer" containerID="e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344" Jul 30 21:20:24.507760 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.507739 2575 generic.go:356] "Generic (PLEG): container finished" podID="e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae" containerID="ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f" exitCode=0 Jul 30 21:20:24.507856 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.507779 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" Jul 30 21:20:24.507856 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.507818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" event={"ID":"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae","Type":"ContainerDied","Data":"ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f"} Jul 30 21:20:24.507928 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.507863 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm" event={"ID":"e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae","Type":"ContainerDied","Data":"3ad5e5b836d38d20f9a31aaff41af4974df52883b922dff0d3b14086ae0db873"} Jul 30 21:20:24.517340 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.517321 2575 scope.go:117] "RemoveContainer" containerID="e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344" Jul 30 21:20:24.517652 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:20:24.517631 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344\": container with ID starting with e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344 not found: ID does not exist" containerID="e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344" Jul 30 21:20:24.517722 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.517660 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344"} err="failed to get container status \"e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344\": rpc error: code = NotFound desc = could not find container \"e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344\": container with ID starting with e19d4a228a383f3d9e3da3ac3cd99a5bce919df4c988227d7ec6f1a4ba39e344 not found: ID does not exist" Jul 30 21:20:24.517722 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.517677 2575 scope.go:117] "RemoveContainer" containerID="ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f" Jul 30 21:20:24.526606 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.526585 2575 scope.go:117] "RemoveContainer" containerID="ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f" Jul 30 21:20:24.526992 ip-10-0-140-21 kubenswrapper[2575]: E0730 21:20:24.526965 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f\": container with ID starting with ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f not found: ID does not exist" containerID="ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f" Jul 30 21:20:24.527234 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.526999 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f"} err="failed to get container status \"ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f\": rpc error: code = NotFound desc = could not find container \"ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f\": container with ID starting with ca13a70dafc0d601ef2dd9ce878dc334be8dd13a44ecaef210eeda9d3f4f631f not found: ID does not exist" Jul 30 21:20:24.530546 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.530520 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp"] Jul 30 21:20:24.533196 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.533177 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e226df-557679dcf8-fvmcp"] Jul 30 21:20:24.542950 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.542919 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm"] Jul 30 21:20:24.546753 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:24.546726 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e226df-6bf4bc7d7b-b78cm"] Jul 30 21:20:26.012997 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:26.012962 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de003a9f-19c2-42c3-93ba-a97b658185cb" path="/var/lib/kubelet/pods/de003a9f-19c2-42c3-93ba-a97b658185cb/volumes" Jul 30 21:20:26.013370 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:26.013340 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae" path="/var/lib/kubelet/pods/e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae/volumes" Jul 30 21:20:36.821821 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:36.821780 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-864fc8cdd8-xvkkf_afce1a47-23af-44d9-b6ea-42aa7c60439f/manager/0.log" Jul 30 21:20:37.182104 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:37.181993 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fb4f68b-7grfq_21995964-ee21-4990-886e-9e6c06b5bd0d/copy-manifests/0.log" Jul 30 21:20:37.195470 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:37.195442 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fb4f68b-7grfq_21995964-ee21-4990-886e-9e6c06b5bd0d/manager/0.log" Jul 30 21:20:38.040251 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:38.040221 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29757420-6k5tr_a16fe61d-707f-4b11-afc0-f237b2e6d1ec/cleanup/0.log" Jul 30 21:20:38.153437 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:38.153407 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29757435-6t8xz_6219b979-4c25-4f59-aa34-7e521727bc63/cleanup/0.log" Jul 30 21:20:38.267133 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:38.267104 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9hs7w_a1d8662c-f616-4dac-9323-1baa4820c6c5/postgres/0.log" Jul 30 21:20:40.257114 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:40.257086 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-54ds6_c98bb757-2b33-4ae5-944e-abc2a991b0df/discovery/0.log" Jul 30 21:20:40.733410 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:40.733380 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zxl5g_18c9c9df-de62-402b-9f53-acd7b1ed728a/payload-pre-processing/0.log" Jul 30 21:20:40.857466 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:40.857436 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-69rsg_b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8/payload-processing/0.log" Jul 30 21:20:40.981100 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:40.981047 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c89bb85d8-xz8kr_7c5e8c92-080a-4b16-844a-e3310c5dc43f/router/0.log" Jul 30 21:20:41.461535 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:41.461508 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-kkgtt_d570968d-e30a-48d6-98ec-136bcfd9d030/main/0.log" Jul 30 21:20:41.577445 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:41.577413 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-fhlj5_0afc292b-b2bf-420b-9309-95c3e03f10e9/main/0.log" Jul 30 21:20:50.590814 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:50.590784 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6htjd_19e1b2f5-58e3-423b-80d7-51670c463768/global-pull-secret-syncer/0.log" Jul 30 21:20:50.789384 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:50.789350 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-st6vb_ede1ef48-7bac-40e6-a395-7357ac99d373/konnectivity-agent/0.log" Jul 30 21:20:50.833479 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:50.833450 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-21.ec2.internal_c79bb9ccdd4a990748877883d79006b3/haproxy/0.log" Jul 30 21:20:58.218126 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:58.218094 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-xqtfv_7ba3571c-c95e-4bc3-a238-eee309a1b456/cluster-monitoring-operator/0.log" Jul 30 21:20:58.413279 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:58.413196 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-vv4fj_ee38a94e-1190-4f5f-a4f3-2875d2eae79c/monitoring-plugin/0.log" Jul 30 21:20:58.613540 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:58.613506 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm7rr_4a798888-f74f-444b-815a-1701950a86b9/node-exporter/0.log" Jul 30 21:20:58.634507 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:58.634486 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm7rr_4a798888-f74f-444b-815a-1701950a86b9/kube-rbac-proxy/0.log" Jul 30 21:20:58.658760 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:20:58.658737 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm7rr_4a798888-f74f-444b-815a-1701950a86b9/init-textfile/0.log" Jul 30 21:21:01.121911 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.121879 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:21:01.127148 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.127126 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/3.log" Jul 30 21:21:01.153203 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153168 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v"] Jul 30 21:21:01.153527 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153514 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de003a9f-19c2-42c3-93ba-a97b658185cb" containerName="payload-processing" Jul 30 21:21:01.153574 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153528 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de003a9f-19c2-42c3-93ba-a97b658185cb" containerName="payload-processing" Jul 30 21:21:01.153574 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153543 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c08dcbb-8299-4d13-b23e-3fea12f49117" containerName="revoke-keys" Jul 30 21:21:01.153574 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153548 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c08dcbb-8299-4d13-b23e-3fea12f49117" containerName="revoke-keys" Jul 30 21:21:01.153574 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153556 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae" containerName="payload-pre-processing" Jul 30 21:21:01.153574 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153562 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae" containerName="payload-pre-processing" Jul 30 21:21:01.153767 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153621 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d5b1c2-2dd8-4157-b2ca-bf485cb355ae" containerName="payload-pre-processing" Jul 30 21:21:01.153767 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153631 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="de003a9f-19c2-42c3-93ba-a97b658185cb" containerName="payload-processing" Jul 30 21:21:01.153767 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.153643 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c08dcbb-8299-4d13-b23e-3fea12f49117" containerName="revoke-keys" Jul 30 21:21:01.156636 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.156618 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.159112 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.159091 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dkmk7\"/\"openshift-service-ca.crt\"" Jul 30 21:21:01.159250 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.159153 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dkmk7\"/\"kube-root-ca.crt\"" Jul 30 21:21:01.159250 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.159174 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dkmk7\"/\"default-dockercfg-xzmq2\"" Jul 30 21:21:01.167611 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.167587 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v"] Jul 30 21:21:01.260792 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.260753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-podres\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.260792 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.260796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7x46\" (UniqueName: \"kubernetes.io/projected/25335606-bfae-48c5-a961-efc1b482bb68-kube-api-access-j7x46\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.261099 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.260854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-sys\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.261099 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.260974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-proc\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.261099 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.261006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-lib-modules\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362400 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-proc\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362400 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-lib-modules\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362642 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-podres\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362642 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7x46\" (UniqueName: \"kubernetes.io/projected/25335606-bfae-48c5-a961-efc1b482bb68-kube-api-access-j7x46\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362642 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-proc\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362642 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-sys\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362642 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-sys\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362642 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-lib-modules\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.362642 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.362603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25335606-bfae-48c5-a961-efc1b482bb68-podres\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.370714 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.370690 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7x46\" (UniqueName: \"kubernetes.io/projected/25335606-bfae-48c5-a961-efc1b482bb68-kube-api-access-j7x46\") pod \"perf-node-gather-daemonset-2xg4v\" (UID: \"25335606-bfae-48c5-a961-efc1b482bb68\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.468600 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.468562 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:01.614662 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.614629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v"] Jul 30 21:21:01.616103 ip-10-0-140-21 kubenswrapper[2575]: W0730 21:21:01.616076 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod25335606_bfae_48c5_a961_efc1b482bb68.slice/crio-0ebc38ab6a910fb16b8b19ba0adcae4b39e2036455fd860de8791be3e800f05f WatchSource:0}: Error finding container 0ebc38ab6a910fb16b8b19ba0adcae4b39e2036455fd860de8791be3e800f05f: Status 404 returned error can't find the container with id 0ebc38ab6a910fb16b8b19ba0adcae4b39e2036455fd860de8791be3e800f05f Jul 30 21:21:01.723214 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.723127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" event={"ID":"25335606-bfae-48c5-a961-efc1b482bb68","Type":"ContainerStarted","Data":"0ebc38ab6a910fb16b8b19ba0adcae4b39e2036455fd860de8791be3e800f05f"} Jul 30 21:21:01.727745 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:01.727719 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b7d76c748-f7sk8_aafaa4d9-0f42-4436-9ce9-48e328d866b8/console/0.log" Jul 30 21:21:02.373275 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:02.373242 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:21:02.389795 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:02.389772 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4lfdb_6cd73e85-ac41-4b79-aa4d-3fcb090e046e/console-operator/2.log" Jul 30 21:21:02.423391 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:02.423363 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-242bj_0f6c52e7-2883-4cc4-9eca-d526502011a5/volume-data-source-validator/0.log" Jul 30 21:21:02.731150 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:02.731112 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" event={"ID":"25335606-bfae-48c5-a961-efc1b482bb68","Type":"ContainerStarted","Data":"279f306c0190e6e6dbaa6a02d40b01e9b5d3985a46953d5adb5931769489a235"} Jul 30 21:21:02.731305 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:02.731175 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:02.748081 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:02.748007 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" podStartSLOduration=1.747991407 podStartE2EDuration="1.747991407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 21:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:21:02.747047794 +0000 UTC m=+3601.212186143" watchObservedRunningTime="2026-07-30 21:21:02.747991407 +0000 UTC m=+3601.213129736" Jul 30 21:21:02.761119 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:02.761091 2575 scope.go:117] "RemoveContainer" containerID="93cd140bc25bfbbb1540f44c36f6bc1d8594436b76101fffe8a76db38d390c44" Jul 30 21:21:02.769953 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:02.769931 2575 scope.go:117] "RemoveContainer" containerID="515dad15e81565ce5f1982583a64b8932ac70ca99303f60c646ac32ac09a53d8" Jul 30 21:21:03.481044 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:03.481015 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nvthr_277d11be-7c47-46e8-8d2a-5e4016c305a7/dns/0.log" Jul 30 21:21:03.503099 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:03.503048 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nvthr_277d11be-7c47-46e8-8d2a-5e4016c305a7/kube-rbac-proxy/0.log" Jul 30 21:21:03.676661 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:03.676633 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mfzp_ff93e08a-42e5-457b-96c2-a54192e5518b/dns-node-resolver/0.log" Jul 30 21:21:04.255791 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:04.255765 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ktn4b_da966ad2-6153-4de3-8d61-4f80ce2fda88/node-ca/0.log" Jul 30 21:21:05.397975 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:05.397938 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-54ds6_c98bb757-2b33-4ae5-944e-abc2a991b0df/discovery/0.log" Jul 30 21:21:05.511337 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:05.511305 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zxl5g_18c9c9df-de62-402b-9f53-acd7b1ed728a/payload-pre-processing/0.log" Jul 30 21:21:05.543888 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:05.543848 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-69rsg_b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8/payload-processing/0.log" Jul 30 21:21:05.566597 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:05.566566 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c89bb85d8-xz8kr_7c5e8c92-080a-4b16-844a-e3310c5dc43f/router/0.log" Jul 30 21:21:06.425690 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:06.425662 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxt4p_fa8706b5-7f1b-4c67-bf3a-bd0a6c3d9685/serve-healthcheck-canary/0.log" Jul 30 21:21:06.895327 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:06.895296 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-wbkxk_84cde953-2d57-4ce5-8848-8079a5023664/insights-operator/0.log" Jul 30 21:21:06.895984 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:06.895966 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-wbkxk_84cde953-2d57-4ce5-8848-8079a5023664/insights-operator/1.log" Jul 30 21:21:06.988912 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:06.988881 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8s8v_d0ff4c71-e236-4d4b-aab7-22eea0229a1b/kube-rbac-proxy/0.log" Jul 30 21:21:07.011184 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:07.011160 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8s8v_d0ff4c71-e236-4d4b-aab7-22eea0229a1b/exporter/0.log" Jul 30 21:21:07.038197 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:07.038172 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8s8v_d0ff4c71-e236-4d4b-aab7-22eea0229a1b/extractor/0.log" Jul 30 21:21:08.746958 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:08.746931 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-2xg4v" Jul 30 21:21:09.611939 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:09.611907 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-864fc8cdd8-xvkkf_afce1a47-23af-44d9-b6ea-42aa7c60439f/manager/0.log" Jul 30 21:21:09.807007 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:09.806971 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fb4f68b-7grfq_21995964-ee21-4990-886e-9e6c06b5bd0d/manager/0.log" Jul 30 21:21:09.837893 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:09.837864 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fb4f68b-7grfq_21995964-ee21-4990-886e-9e6c06b5bd0d/copy-manifests/0.log" Jul 30 21:21:11.411200 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:11.411108 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-696d745bf6-rntwr_915c3e37-ec0b-4587-b834-ed7e657eebf0/manager/0.log" Jul 30 21:21:12.309351 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:12.309316 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-54ds6_c98bb757-2b33-4ae5-944e-abc2a991b0df/discovery/0.log" Jul 30 21:21:12.384675 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:12.384644 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zxl5g_18c9c9df-de62-402b-9f53-acd7b1ed728a/payload-pre-processing/0.log" Jul 30 21:21:12.403761 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:12.403726 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-69rsg_b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8/payload-processing/0.log" Jul 30 21:21:12.421849 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:12.421816 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c89bb85d8-xz8kr_7c5e8c92-080a-4b16-844a-e3310c5dc43f/router/0.log" Jul 30 21:21:14.053769 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:14.053738 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-54ds6_c98bb757-2b33-4ae5-944e-abc2a991b0df/discovery/0.log" Jul 30 21:21:14.139638 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:14.139611 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zxl5g_18c9c9df-de62-402b-9f53-acd7b1ed728a/payload-pre-processing/0.log" Jul 30 21:21:14.160258 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:14.160231 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-69rsg_b066fe25-58b5-4dd3-838a-2d8c0fa9b4f8/payload-processing/0.log" Jul 30 21:21:14.173342 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:14.173317 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c89bb85d8-xz8kr_7c5e8c92-080a-4b16-844a-e3310c5dc43f/router/0.log" Jul 30 21:21:16.434584 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:16.434550 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-jcgbc_6c6076ec-c274-4e50-ad5c-8dbbfc677104/migrator/0.log" Jul 30 21:21:16.456297 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:16.456272 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-jcgbc_6c6076ec-c274-4e50-ad5c-8dbbfc677104/graceful-termination/0.log" Jul 30 21:21:16.801926 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:16.801896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-xx7xc_a0f10072-fc2c-4d23-91b3-1a982a9decbd/kube-storage-version-migrator-operator/1.log" Jul 30 21:21:16.802697 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:16.802675 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-xx7xc_a0f10072-fc2c-4d23-91b3-1a982a9decbd/kube-storage-version-migrator-operator/0.log" Jul 30 21:21:18.132188 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.132162 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkqt8_41464378-d96a-4890-bcdb-fb23717b7593/kube-multus-additional-cni-plugins/0.log" Jul 30 21:21:18.153911 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.153877 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkqt8_41464378-d96a-4890-bcdb-fb23717b7593/egress-router-binary-copy/0.log" Jul 30 21:21:18.175626 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.175603 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkqt8_41464378-d96a-4890-bcdb-fb23717b7593/cni-plugins/0.log" Jul 30 21:21:18.196949 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.196926 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkqt8_41464378-d96a-4890-bcdb-fb23717b7593/bond-cni-plugin/0.log" Jul 30 21:21:18.218710 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.218686 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkqt8_41464378-d96a-4890-bcdb-fb23717b7593/routeoverride-cni/0.log" Jul 30 21:21:18.239810 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.239781 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkqt8_41464378-d96a-4890-bcdb-fb23717b7593/whereabouts-cni-bincopy/0.log" Jul 30 21:21:18.261351 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.261320 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkqt8_41464378-d96a-4890-bcdb-fb23717b7593/whereabouts-cni/0.log" Jul 30 21:21:18.302799 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.302771 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l85qb_4343dc18-e764-4840-a67e-fa8389bf80eb/kube-multus/0.log" Jul 30 21:21:18.441435 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.441355 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dgsbq_dfc8c488-8fee-4f10-a041-8d67bc0569a8/network-metrics-daemon/0.log" Jul 30 21:21:18.462522 ip-10-0-140-21 kubenswrapper[2575]: I0730 21:21:18.462488 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dgsbq_dfc8c488-8fee-4f10-a041-8d67bc0569a8/kube-rbac-proxy/0.log"